Hello everyone, and welcome. My name is Gupta and I'm excited to be speaking with you today at Con 42, we are going to dive into a critical and rapidly evolving area resource allocation for AI workloads in cloud computing. The rise of AI applications from deep learning models to complex machine learning pipelines has completely changed the game for cloud resource management. These aren't your traditional applications. They have unique demands, complexities, and behaviors that require much more sophisticated strategies for allocating compute, memory, and network resources effectively as we'll. Explore AI workloads are fundamentally different. They exhibit significant resource variability, often requiring specialized hardware like GPUs. Or TPOs. Their performance is highly sensitive to latency and throughput, and their resource needs can change dramatically depending upon the face of execution. Think training versus inference. These unique characteristics create novel challenges that traditional cloud management systems often struggle with. This talk, we will examine the convergence of AI and cloud infrastructure. Looking critically, how can we effectively manage resources in this new landscape? So let's ask this question. Why is this so important? Look at the scale and impact. The global AI infrastructure market is projected to reach a staggering 2 99 $0.64 billion by 2026. That's not just not, that's not just growth, it's explosive growth. Their annual growth rate at 35.6% is significantly outpacing that traditional IT sectors, CAGR since 2021. This signifies a massive shift in investment and focus towards AI capabilities. But it's not just about spending more, it's about spending smarter. Implementing AI driven resource management itself feels significant. Benefits. Industry analysis show that these intelligent systems can lead to transformative improvements. We are seeing potential for up to 40% improvement in resource utilization, meaning less wasted capacity through techniques like ML powered allocation. Simultaneously, organizations are documenting 25 to 35% reduction in operational cost, primarily infrastructure expenditure, while maintaining strict performance banks, benchmark and service level agreements. Status. SLS, this dual optimization, improving efficiency and reducing cost represents a true paradigm shift in cloud economics Driven by the very AI technologies we are trying to support. To allocate resources effectively, we first need to understand what the workload needs. This is where workload profiling and resource estimation comes in. How much C-P-U-G-P-U memory and IO will a specific AI task require, and when we have several approaches. First static analysis. This involves looking into the AI model's architecture. Its hyper parameters and known pattern before execution. For established workloads, this can predict computational demands with reasonably high precision, around 80 to 85%. Second, historic pattern analysis. Here we analyze past execution, data telemetry to identify temporal signal signatures. Advanced time series modeling and techniques like wave decomposition help forecast future requirements based on past behavior. Third, online monitoring. This is about real-time assessment. We continuously sample critical system matrix during the runtime itself. Things like compute, saturations, memory, bandwidth usage, and IO throughput to understand the immediate needs and dynamically adjust. Last, but not the least. Hybrid approaches. Often the best strategies combines these three methods. We integrate complementary techniques to enhance prediction accuracy. However, it is important to note that for completely new and unpredictable workload patterns, accuracy can degrade perhaps for the 60 to 65% range. Getting this profile right is crucial. As noted, significant workload intelligence systems have led to the infrastructure cost reductions of up to 35% and utilization improvements of 40%. The accuracy of our profiling directly impacts how efficiently and stably we can operate these next AI workloads. Okay, so once we have an estimate of what the resources need, how do we actually assign those resources to the workloads waiting to run? This is the job of the scheduler for ai. We need intelligence scheduling mechanisms. First priority based. Not all workloads are equal. ML algorithms can analyze patterns and history to anticipate needs and inten intent and intelligently preempt lower priority tasks to ensure critical jobs get the resources they need when they need them. Second, fair share. In multi-tenant environments, we need fairness. Sophisticated frameworks ensure equitable resource distribution across different users or groups while still aiming for the overall system efficiency. Deadline aware. Many AI tasks, especially in production pipelines have critical time constraints. The scheduler uses predictive model that factor in computational complexity, data throughput, and dependency to ensure tasks complete within their deadlines. Optimization resource allocation is fundamentally a complex, multi-dimensional optimization problem. Advanced approximation algorithms are used to find computationally feasible solutions that are close to the optimum. Even if finding the perfect solution is intractable, often an NP hard problem. Despite these advancements, the inherent computational complexity, the NP heart nature of the optimal multidimensional allocation means we often rely on design heuristic approaches. This aim to balance practical real world performance with theoretical ality guarantees it's a constant trade off. Okay, so AI workloads are rarely static. Their demands fluctuate. Autoscaling is the key to handle this dynamic efficiently. A typical AI powered autoscaling architecture involves several layers. First monitoring layer, this is the foundation. It collects detailed telemetry data across multiple dimensions. Your CPU, your GPU Memory network application specific matrix, providing a rich granular view of the system, state and resource utilization. Sophisticated deep learning models are often used to analyze these complex patterns. Second layer is your analysis engine. This is the brain. It implements pattern recognization and decision models often leveraging reinforcement learning algorithms. It. Continuously evaluates historic performances and real time systems state to decide when and how to adjust resources. Should we scale up, scale down, or stay Put. Third execution layer. This layer translates the decision made by your analysis engine into the concrete actions for your infrastructure. Provisioning new virtual machines, adjusting container replicas, modifying resource limits. While these AI powered auto-scaling systems showed remarkable capabilities in control tests achieving prediction accuracy of often around 95%, they face significant challenge in the real world production. We confronted sorry. When confronted with unexpected traffic patterns, anonymous behaviors or system events. Prediction accuracy can deteriorate significantly. Sometimes it drops to as low as 72%. This highlights the ongoing need for robustness and better handling for unseen for circumstances. Autoscaling Achieves resource elasticity, the ability to adjust resources dynamically. This is implemented primarily in two ways. Vertical scaling and horizontal scaling. Let's first talk about vertical scaling. This means adjusting the resources allocated to the existing instances. Think adding more CPU or GPU or memory to our running virtual machine. Using GPU partitioning technologies to allocate fraction of A GPU, employing memory ballooning techniques without super within hypervisors. These limitations are often the limitations here are often physical hardware constraints, operating system support and application compatibility. Not all application can handle graceful and resource changing underneath them. Second is horizontal scaling. This involves adding or removing entire instances. So remember, vertical scaling means adjusting the resources within the existing instances. Horizontal scaling, we add or remove the entire in instances common mechanism of include auto scaling groups that can manage a pool of identical instances, replica controllers in orchestration systems like Kubernetes, and load balancing mechanisms to distribute traffic across the available instances. The main challenge here is the coordination overhead. As you scale up out to many instances, managing them and ensuring that they work together efficiently introduces complexity and throughput gains can diminish beyond a certain point. These elasticity mechanisms are absolutely critical for modern ai. Especially generative AI workloads that we have today, which can exhibit dramatic peak andro in resource demands during different processing or phasing slides. Okay, so now let's talk about this. Containers have become a standard way to package and deploy applications, including AI, workloads, orchestration platforms, manage these containers at scale. Kubernetes has largely emerged as the prominent orchestration platform for ai. It provides a robust framework with several key components, networking layer, storage, layer data, plane and control plane. Your networking layer facilitates seamless communication between containers, often using sophisticated overlay networks. Your storage layer ensures data persistence across container life cycles crucial for stateful AI applications. Often using dynamic volume plugins, data plane, this is where the work happens. Components like the ber culet on each note coordinate with container run times to execute the workload control plane. The brains of the operation, the API Server Intelligence Scheduler, and Robb Robust controller managers orchestrate the entire system. Empirical studies consistently show Kubernetes outperforming alternates like Docker Swamp in large scale deployments, managing thousands of nodes and containers efficiently, while maintaining high resource utilization and workload throughput. However, this comes at a cost. Kubernetes introduces significant architectural complexity and operational challenges. Organization needs specialized expertise and comprehensive training. It's a high barrier to entry compared to Simpl solutions, but it ultimately delivers the superior scalability and flexibility needed for the complex AI computing environments. Underlying much of cloud computing and resource allocation is virtualization, abstracting the physical hardware. Several techniques are vital for ai. CPU virtualization techniques like Intel VTX and A MDV provide hardware assistance to reduce the overhead of virtualizing the CPU, enabling near native performance for compute intensive tasks, memory virtualizations technique like SLAT, second level address translation, numa, non-uniform memory access. Awareness in the hypervisor and transparent pace sharing improves memory efficiency and introduce across latency. Critical for data intensive AI applications. Input output virtualization. Getting data in and out quickly is crucial. Technologies like single route I virtualization, para virtualization drivers, and direct di direct device assignment, bypass some of the virtualizations over it for network and storage operations. Enhancing performance GPU virtualization. Essential for many AI workloads method range from AI remoting to hardware assisted partitioning, and time slicing mechanism that allow multiple concurrent users or workloads to share a single physical GPU. It is important to add a caveat here. While these advancements offer real native while these advancements offer near native performance under optimal conditions, or for a specific workload types, performance degradation can still be significant, particularly for applications that are highly IO intensive. The virtualization layer, however, thin still introduces some overhead. Ultimately, a major driver for sophisticated resource allocation is cost optimization. How can we run these expensive AI workloads more economically? Several strategies are the key here. First, VM allocation policies intelligently allocating virtual machine resources. Based on the actual need rather than over promising can yield significant savings, potentially 30% cost reduction workload placement, strategically distributing workloads across different availability zones or even geographic regions. Based on the resource availability and pricing difference can achieve a saving up to 25% resource right sizing this wind bots precisely matching the allocated resources. CPU memory, GPU to the actual measured workload requirement, avoiding any waste. This can contribute another 20% reduction. Commitment discounts. Cloud providers offered substantial discounts, offered 45% or more for long-term commitments, like reserved instances or saving plans compared to on demand pricing. Leveraging these for a baseline workload is crucial. Traditionally, traditional cloud environments often operate as a surprisingly low efficiency, maybe 30 to 45%. Implementing these optimized allocation policies can dramatically increase utilization rates, pushing them towards 65 to 75%. However, these figure often represent idolized scenarios where organizations can have complete flexibility. Real world constraints can limit the achievable sale. Achievable savings. Okay, so shifting resources dynamically in multi-tenant environment introduces unique security consideration. Sharing isn't always caring when it comes to sensitive data and workloads. Key security aspects include. Sorry. Isolation mechanism. Strong boundaries are essential Hypervisor. Security features and kernel level isolation prevent unauthorized access between different workloads running on the same physical hardware authorization systems. Robot robust policy engines must govern who or what can make resource allocation decisions adhering to the principle of least privilege, continuous validation. Security isn't a one-time check. We need continuous monitoring of allocation decisions against the security policy for defense in depth site channel protection. When multiple tenants share physical resources, especially accelerators like GPUs, there's a risk of information leakage through subtle timing variations or cache aspect cache access patterns. Mitigation against these attacks are crucial. So container platforms like Kubernetes also presents with their own security challenges when used for AI hardening measures such as restrictive port security policies, network policies to control traffic flow and runtime security monitoring are necessary additions. Security must be baked into an allocation strategy, not bolted on afterwards. Okay, so now if we have come to our end of the presentation, let's talk about where is this field heading. Several exciting directions are emerging. First, automated workload characterization using AI itself, deep learning, transfer, learning casual inference to automatically fingerprint, workloads, adapt quickly to new application types. And pinpoint resource bottlenecks without manual analysis, energy aware allocation, implementing scheduling, and scaling policies that are conspicuous of power consumptions. This includes dynamic voltage, frequency scaling and consolidating workloads based on thermal characteristics to reduce the overall energy footprint. Carbon aware computing, taking energy awareness a step further by scheduling workloads to align with the availability of clean energy. This involves integrating with grid carbon intensity forecast, and renewable energy production patterns to run computation when it is the greenest. In conclusion, integrating AI into resource management creates a fascinating meta recursive system, AI optimizing the infrastructure needed to run ai. This enables unprecedented automation and efficiency, but also introduces novel complexity and challenges. As we have discussed, this technical analysis that we have done today highlights the critical need for continued research. We need to address Al, sorry. We need to address algorithmic limitations, improve the robustness of these systems, especially when facing the unexpected, undeveloped, standardized benchmarking methodologies. To objectively compare different approaches. The journey of efficiently powering the AI revolution is still very much under the way. I hope this presentation helped you in a way to understand it. Thank you.