For being here, I'm Pril Marni, senior DevOps professional with around 10 years of experience. Today we are going to talk about harnessing Kubernetes for scalable AI ML workloads, using insights from Tesla and open ai. Over the next few minutes, we'll explore why Kubernetes has emerged as a defacto platform for container orchestration in ai, and we'll dive into two real world implementations that leverage Kubernetes for AI ML workloads. I share architecture patterns, operational metrics, and lessons learned that you can apply in your own environments. First up, let's discuss how AI workloads differ from traditional computational workloads. AI workloads, as are resource intensive, require elastic scaling and are failure prone when thousands of GPUs are pushing petabytes of data. 24 7 analysts project the market for AI platforms will search from five $15 billion in 2023 to over $2 trillion by 2032. So the stakes are huge here. Traditional static clusters can't keep up. And based on recent surveys, 61% of the enterprises site capacity bottlenecks as their number one blocker to production. Ai. Considering this, Kubernetes has emerged as a critical solution for addressing these challenges through dynamic resource allocation, intelligence, scaling self-healing capabilities, enhanced monitoring and workload portability. Coming to some of the common AI infrastructure challenges, resource intensity is one of the top challenges faced by AI infrastructure. Modern AI models demand extraordinary computational resources with training requirements increasing exponentially modern AI training and inference pipelines demand a compute. Driven model training can consume over 61% of the data centers total power budget, leaving less headroom for networking, storage and non-AI workloads. Scaling complexity, data volumes and model par parameter counts are exploding at around a 36% compound annual growth rate. This means a cluster sized for last year's workload will be under provision today and obsolete tomorrow. Static infrastructure simply can't keep up with the pace of growth that we have seen here. Resource inefficiency. Without proper orchestration, organizations struggle to optimize resource allocation across varied workloads with fluctuating demand patterns. As pipelines become more distributed, spanning object stores messages, andous compute failure modes multiply and increase potential points of failure. Without automated recovery, mean time to repair can stretch into hours or even days. This is extremely concerning in large scale distributed training environments. These challenges together combine to slow down, model iteration, inflate costs, and undermined re reliability. Next, let's see how Kubernetes directly addresses each of these issues as a solution. Alright, let's see how Kubernetes as a solution for us here. According to CNCF research, 88% of organizations are running containerized AI ML workloads in production with 70% leveraging Kubernetes as their primary orchestration platform. This widespread adoption is driven by Kubernetes, is comprehensive feature set that directly addresses the unique requirements of AI workloads. Kubernetes comes with declarative provisioning. In a single yaml, you get to describe GPU type. The driver, CUDA base image and Kubernetes device plugin automates the rest. No more handbuilt bare metal clusters. It has great resource optimization and scalability. It provides tools like horizontal power autoscaler, which can combine with NVIDIA metrics exporter. Allows Kubernetes to adjust replicas in mere seconds while vertical port or autoscaler right sizes, memory, or CPU based on the requirement. CADA adds event driven bursts that can help with sudden spikes in traffic. Kubernetes has self-healing and rolling updates for an improved uptime. If A GPU node panics the scheduler, restarts the pod on healthy hardware and respects pod disruption budgets. So training jobs keep their core. If any update needs to be deployed to the model, Kubernetes control loop based resource management ensures that the new changes are rolled out without any downtime. Kubernetes offers multi-tenant isolation using namespace resource quotas, thas and node selectors. It lets you guarantee say that 40 a one hundreds are for team A and 40 H. One hundreds are for production inference. You can divide your own GPU hardware. Allocate specific services, specific workloads onto each type of hardware. It also enables for batch AI workloads to be deployed onto selective infrastructure. So to summarize, Kubernetes provides dynamic resource allocation, allowing organizations to optimize resource utilization across variable AI workloads. As you can see from CNCF research, these percentages are not a smaller percentages. They reflect broader industry adoption. Underscoring that Kubernetes is production hardened for ai. Okay, let's continue our discussion about Kubernetes capabilities for AI workloads. Let's dig deeper into how each functionality helps with. Running AI workload dynamic resource allocation. In the previous slide, we had already covered a little bit about how Kubernetes allows for enhanced resource optimization and scalability through the GPU device plugin and custom scheduler extensions. Kubernetes can share GPU devices across parts or carve them into partitions. In practice, 63% of the organizations citing resource optimization as a primary motivation for adopting Kubernetes intelligent scaling. Kubernetes provides multiple scaling mechanisms that address variable resource requirements. The horizontal pod autoscaler and vertical pod autoscaler augmented with custom metrics such as GPU, memory pressure or Q Depth allow training and inference pods to scale up and down in real time, which such scaling 78% of the organization are to adopt into ku. To self-healing capabilities of Kubernetes, Kubernetes provides critical reliability improvements like liveliness probes, readiness probes that can catch hunger or crash processes. Cluster autoscaler can be used to replace unhealthy nodes automatically without any manual intervention. A 72% reduction in failed training runs across organizations. Kubernetes provides enhanced monitoring. Kubernetes offers granular visibility into resource utilization and performance metrics. It's out of the box. Integration with Prometheus and Grafana provides dashboards for GPU, temperature power draw, PCIE. Throughput and network IO teams achieve 76% faster. Mean time to recovery because anomalies are detected and visualized immediately. These aren't marketing claims that I was just showing you. They're aggregated from production telemetry across hundreds of deployments. That said, let's continue on to our case study of one of the prominent car manufacturers innovations Tesla. Tesla represents one of the most sophisticated implementations of Kubernetes for AI ML workloads, leveraging container orchestration to power. Its autonomous driving technology. The company's autonomous driving system processes data from eight cameras that collectively capture 360 degree video generating approximately 1.5 terabytes of data per car annually. Tesla processes over a hundred thousand video clips per day through its computer vision pipelines. With each clip requiring analysis across multiple neural networks, Kubernetes orchestrates thousands of container instance instances that collectively analyze these inputs during training period. By contain their 360 degree camera inference, pipeline deployment time from two weeks to under four hours. Tesla employs a hybrid cloud approach for its training infrastructure, which is very important. Cloud brings you that reliability and flexibility, whereas On-Prem gives you that high performance with Kubernetes, managing workloads across both on-premise data centers and cloud resources for from multiple providers. The hybrid approach did enable Tesla to optimize for both cost and performance. Let's see how they have built their Kubernetes implementation. Let's go through each component and see how each component serves which purpose in their whole architecture, AI m pipelines. Tesla used PY and TensorFlow for their AI ML pipelines to to do neural network training and used Triton inference server for their realtime inference. And all of these were running inside Kubernetes parts and the time inferencing for each camera orientation is done using these AI ml. Infrastructure approach is hybrid cloud, as we already discussed with on-prem and cloud to optimize cost and performance coming to hardware accelerators, Tesla used Nvidia GPUs and some other custom AI chips built for specific use cases. And these are used to power their neural network training model training and inferencing. The training technique they adopted is data parallelism. And they have achieved this by distributing workloads Kubernetes into Kubernetes across multiple GPUs, and they leverage Kubernetes to achieve this. Deployment mechanism they have a unique style of deployment mechanism compared to traditional use cases. Since they have a fleet of cars that have to function and do real-time inferencing, they use something called over the air updates to deliver their model improvements. Pull new container images and rotate parts seamlessly. So every car pretty much runs their latest neural net weights. The architecture, this architecture that they have employed. Decouples model development from infrastructure, accelerating Teslas iteration route. Now let's discuss the case study of one of the cutting edge AI platform. That brought this whole AI frenzy to the market open ai. Open AI represents a prime example of how Kubernetes can be leveraged to manage extraordinary computational demands of cutting edge AI research training. Modern LMS like those developed by open AI requires massive computational resources. GT three featuring one 75 billion parameters and G PT four estimated to have more than 1 trillion parameters. Kubernetes provides open AI with the ability to define sophisticated scheduling rules that consider complex variables like data locality. Interconnect bandwidth and power constraints. The platform's native support for GP resources allows for precise allocation of specialized computing resources. Kubernetes provides distributed training, so open AI leverage HO Award on Q Flow, which runs on Kubernetes, and they used it to orchestrate data parallelism to reduce. Their workloads across hundreds of GPU parts. Data processing Spark on Kubernetes was used by OpenAI to handle ETL of petabytes of text tokenization, filtering, and feature extraction. For model serving, they used Triton plus custom Kubernetes operators and use them to route thousands of inference request per second. Sub 50 millisecond latency, which is pretty low. And this was only possible because of using Kubernetes. Now, let's see what other capabilities that OpenAI has leveraged using Kubernetes. As you all know, as we already discussed, the resource management is one of the key aspects that Kubernetes triumphs over. OpenAI has leveraged Kubernetes for their GPU resource management for efficient allocation of hundreds of thousands of GPUs for distributed LLM training, optimizing them for performance and cost. For example, their high priority training jobs preempt lower priority workloads, ensuring SLAs for critical research runs. So there are high priority training jobs run on parts that get scheduled on GPUs that are more powerful that way. Kubernetes gives you that flexibility of resource management to ensure performance and efficiency. Swarm based ML orchestration. Coordination of multiple AI agents working on different aspects of the MI ML pipeline. Breaking down the training process into discrete containerized steps is key for open ai. Custom controllers treat hundreds of parts as a cohesive form, packing GPUs to maximize efficiency. Dynamic scheduling is one of the important aspects for achieving higher throughput with lower latency. Think of it, adjusting resource allocations based on changing requirements across different training phases from CPU intensive pre-processing to intensive training. Parts can migrate between CPU only ETL stages and GPU accelerated training inferences stages based on demand and can be scheduled onto specific type of node. This dynamic scheduling scale up and scale down aspects of Kubernetes, which is automated, gives open ai. Edge over the other AI platforms. This Edge made them choose Kubernetes over as their primary infrastructure platform for ai. Okay, now the last aspect is monitoring and observability. Who doesn't need monitoring across their workloads? Be it traditional workloads or AI workloads. Monitoring and observability is a key aspect for maintaining uptime and product efficiency. Tracking resource utilization, model performance, and system health across distributed infrastructure to identify bottlenecks and anon anomalies is key for any product lifecycle. They have implemented distributed tracing of RPCs, GPU Sims, SM metrics and Network Telemetry triggering alerts in under a second for anomalous patterns. This gives them that edge that can help with monitoring their AI workload. These capabilities altogether allow open AI to operate at a scale few organizations can match, and they have achieved this high growth using Kubernetes while maintaining high utilization and reliability. Alright, let's discuss. AI Ecosystem Q is one of the end-to-end platform for orchestrating sophisticated ML pipelines provided, providing streamlined model training and hyper parameter tuning using CIB and metadata tracking with ML MD and it is. Very important tooling used by many organizations for their production deployment workloads. Coming to TensorFlow operators, these are custom Kubernetes controllers that auto automate TensorFlow distributed training configuration, dramatically simplifying resource allocation and internode communication. TensorFlow operators and media device plugins. Natively expose GPUs and multi-instance GPUs to parts coming to model serving production. Grade inferencing frameworks like Kerv, kf serving seldom core and Triton deliver auto scaling, canary rollouts and AB testing. For inference. They provide auto scaling and deployments for seamless AI delivery. Importantly, Kubernetes provides enhanced security frameworks using, which are S-P-I-F-F-E, SPIRE for Workload Identity, GRPC, mutual TLS, and Network policies for Zero Trust. They also provide advanced isolation technologies. Like Carta containers that create hardware virtualized environments for high value AI models, protecting intellectual property and sensitive data altogether, the Kubernetes ecosystem has matured into a comprehensive AI ML platform, evolving from basic container management to offering specialized tooling that addresses the entire machine learning or lifecycle with enterprise adoption. Accelerating the global cloud native platforms market is projected to reach $62.7 billion by 2034, growing at A-C-I-G-R of 16.5%. As organizations increasingly leveraged these technologies for competitive advantage. Together. These projects fill every stage of the AI lifecycle from data ingestion to model deployment, all taken care of within Kubernetes. Alright, let's look at some of the common industry adoption in Kubernetes for AI workloads. Industries are adopting cloud native platforms like Kubernetes at different kinds of rates with healthcare and financial services leading the way. Healthcare is currently sitting at 18.2 CIGR Genomic sequencing pipelines and MRI Ima image analysis are running on Kubernetes clusters to achieve high healthcare needs. The healthcare sector's adoption is particularly notable because of the stringent regulatory requirements and sensitive patient data. Banking, financial services and insurance are sitting at 17.9% annual growth rate. Growth driven by driven the need for secure AI infrastructure with technologies like Carta containers that we have already discussed about, has been especially valuable for processing sensitive financial data and algorithmic trading strategies. Finance industry has seen a rise in AI adoption for realtime fraud detection using streaming inference, and Kubernetes has been. Telecom around 0.8 annual growth. It is leveraging Kubernetes to manage complex networks and deliver innovative digital services at scale with high reliability. For example, one of the top use cases for telecom is to achieve superior 5G network analytics and edge AI in micro data centers powered by Kubernetes. Coming to manufacturing. It's currently sitting at. But the experts are expecting that this could be become one of the prime sector for AI adoption and that AI adoption calls for Kubernetes adoption. So implementing Kubernetes to orchestrate iot devices, optimize production lines, and enabling predictive maintenance using AI systems. Which run on Kubernetes has been the has been the motive for manufacturing growth rate, AI driven visual inspection, and predictive maintenance via GPUA. Accelerated vision models that run on Kubernetes is another top use case that drives this adoption. Alright, let's wrap up and discuss what we have already gone through in the last few minutes. Kubernetes provides great infrastructure foundation for scalable reliable AI infrastructure. It delivers the elasticity, resilience and portability that modern AI ML workloads demand. Kubernetes has an expanding ecosystem that has specialized tools for enhancing capabilities for AI specific requirements, and it is ever evolving. C and vendor extensions continue to evolve. Filling gaps in security, pipeline orchestration and serving. Accelerating innovation is one of the top drivers in Kubernetes and top requirements for AI organizations gain try to gain competitive advantage through improved resource utilization. By standardizing on Kubernetes teams shift their focus from infrastructure, plumbing, to model development and data science. As AI models grow in size and complexity, Kubernetes will remain the orchestrator of choice, enabling continuous experimentation, rapid ation, and cost efficient operations. Alright. Thank you for the attention. I hope these insights help you architecture and operate your own Kubernetes based AI platforms. And they, I hope that they gave you a good depth, in depth look at how Kubernetes can help with running AI ML workloads. Thank you. And have a.