Hello, good morning. I am de Kana a senior data engineer with experience in cloud native infrastructure and data platforms. My motivation for this research came from seeing firsthand how cloud workloads are growing more dynamic and complex, especially in large scale environments like retail and e-commerce. I have worked on projects where traditional Kubernetes resource management simply couldn't keep up with unpredictable traffic and changing business needs. This inspired me to look for smarter and AI driven solutions. Let me share a quick story. In one of our retail systems, we experienced a certain spike in online traffic during a flash sale. Our Kubernetes cluster was set up with Static Horizon and report auto scheduler thresholds. The static policies couldn't react fast enough. Some services were well provisioned based in cloud resources, while others were under provision causing slowdowns and even outages. Manual intervention was needed to adjust configurations, which isn't capable, scalable, or reliable in fast moving environments. This is highlighted the limitations of static resource management. Today I'll show you how deep reinforcement learning can help Kubernetes clusters learn and adapt in real time, making smarter decisions about resource allocation. We'll see how AI can recognize patterns in workload behavior, optimize for cost and performance, and reduce the need for manual tuning, et cetera. By the end of this talk, you'll understand how intelligent automation can futureproof your cloud infrastructure and unlock new efficiencies. Let's proceed today. I outlined my presentation in such a way that to take you on a journey where we'll start by understanding the core challenges in Kubernetes resource management. Then move to why deep reinforcement learning is promising solution. Next, I'll share a research foundations and specific optimization areas we targeted, followed by look at the RL algorithms we used and how we implement them. We'll discuss the unique challenges of multi cluster environments, the C-N-F-C-F tools to support our approach and how we validated our results through benchmarking. Finally, I will cover production deployment considerations and outline practical next steps so you can see how these ideas translate into real world impact. While we dive into some technical details like algorithms and integration strategies. I'll make sure to make you better understanding on that. Whether you are a data scientist, DevOps engineer, or a cloud architect, you'll live with both a deeper understanding of the technology and concrete ideas where you can apply in your own environments. Let's discuss about the challenges. Let's imagine a scenario. Your company launches a new product and suddenly thousands of user floods your website. The Kubernetes cluster configured with static resource limits can't scale up fast enough. Some services crash due to lack of resources while others are running with excess capacity leading to wasted cloud spend. This is inges a theoretical problem. It's something many organizations face during peak events, sale or unexpected viral moments. When these pipes happen, DevOps team are forced to jump in and manually adjust configurations often late at night and during critical business hours. This firefighting takes time away from innovation and strategic work and increases the risk of human error. Manual tuning is not only stressful, but also unsustainable. As system grow more complex and workloads become unpredictable. These challenges are widespread, and they're exactly what we are aiming to solve with this intelligent and adaptive solutions. Now, why deep reinforcement learning traditional orders curling in Kubernetes relies on fixed thresholds and reactive policies? When a metric crosses a certain timeline, the system responds, but it doesn't anticipate or leave from past, learn from past behavior. In contrast, deep reinforcement learning enables Kubernetes to continuously learn from its environment. Our agents observe the pattern, predict future needs, and proactively adjust resources. They shift from reactive to adaptive management, means cluster can handle unpredictable workloads more efficiently with less manual intervention. RL algorithms excel at uncovering complex relationships in data. Patterns that are often too subtle are multidimensional for humans to spot. For example, a L can detect cyclical traffic spikes, seasonal trends, or correlations between different microservices that static rules would overlook. By learning from historical data and realtime feedback, AL can optimize resource allocations in a ways that manual tuning simply can't achieve it. Reinforcement learning has already revolutionized fields like robotics, where agents learn to navigate complex environments and gaming. Where AI has mastered games like Go and StarCraft, these successes show RLS power to solve dynamic, high dimensional problems just like those we face in cloud infrastructure. By bringing R to Kubernetes, we are leveraging proven AI techniques to make our clusters smart. More resilient and more cost effective. Now, let's discuss on some research foundations. Our approach was inspired by several pioneering studies in the field of cloud resource optimization using reinforcement learning. For example, research from leading universities and tech companies has shown RL can outperform traditional auto scaling in dynamic environments. Benchmarks from these studies demonstrated significant improvements in CP utilization and response times, which motivated us to explore RL for Kubernetes workload management. We also looked at open source RL frameworks and their application in real world cloud scenarios to guide our methodology. One of the most powerful aspects of RL is its ability to continuously learn and adapt. Unlike static policies, RL agents evolve as workloads and infrastructure change. This means your Kubernetes cluster can stay optimized even as new applications are deployed. Traffic patterns shipped or hardware is upgraded. Continuous learning helps future proof your infrastructure and reducing the need for frequent manual reconfiguration and keeping your systems resilient to change. During our experiments, we found that RL driven clusters not only improve. Resource utilization, but also reduce the number of scaling events, making resource management more stable and predictable. One. A surprising result was the ability of RL agents to anticipate traffic surges before they happened. Thanks to the pattern recognition in from historical data. In literature, there are cases where RL reduced cloud costs by up to 40% compared to traditional autoscaling highlighting the real world impact of intelligent optimization. So what are the core optimization areas? Let's break down the core optimization areas where machine learning can make a real impact. First POD scheduling. For example, ML can predict which notes will have the right resource available and place pods accordingly. Reducing bottlenecks and improving performance resource allocation. Instead of fixed to CPU and memory assignments, ML models can dynamically adjust resource based on predicted demand preventing both W and under provisioning traffic routing. In a service, mesh Amil can optimize routing decisions in real time, sending requests to the healthiest or most cost effective endpoints. Multi cluster orchestration, Amil can help distribute workloads across clusters in different regions or clouds, balancing load, and minimizing late density by automating complex decisions. ML driven optimization reduces the need for manual tuning and constant oversight. This means DevOps team can focus on strategic tasks rather than fire fright, resource issues. Efficiency improves because resources are allocated precisely when and where they are needed. Minimizing, wasted and maximizing performance, real-time feedback is crucial for effective optimization. ML models rely on continuous monitoring of cluster metrics and application performance. This feedback loop allows the system to quickly adapt to changing conditions such as traffic spikes or hardware failures. Ultimately, realtime feedback ensures that optimization decision remain in relevant and effective even as workloads evolve. Let's see how deep Q networks for containers orchestration. DQ networks or dqn are particularly effective for problems where decisions are discreet, such as choosing which node to place a PO or selecting a resource tire. In Kubernetes, many resource management actions are not continuous, but involve clear choice. DQN can evaluate the current state of the cluster and select the best action from a set of possibilities. This makes DQNA natural fit for orchestration, container placement, and scaling decisions in a dynamic environment. One of the biggest challenges in applying DQN to Kubernetes was designing a reward function that truly reflects the business goals. For example, we had to balance performance, cost, and reliability. If the reward only focused on CP utilization, it might ignore response time and efficiency. Cost efficiency. We experimented with multi object to reward functions and found that waiting different metrics appropriately was key to achieve the decide outcome experience. Relay is a technique, sorry. Experience replay is a technique where the RL agent stores past experiences and samples then randomly during training. This helps break the correlation between co experiences making learning more stable and robust. Especially in dynamic environments like Kubernetes, where conditions can change rapidly. By revisiting a diverse set of scenarios, the agent learns more generalizable policies and avoids over fitting to the recent events. Proximal policy optimization in action PPO, it is also known for its stability during training, which is crucial when making continuous resource allocation decisions in Kubernetes. Unlike algorithms that can make abrupt changes, PPO uses a clip objective function to ensure updates to the policy are gradual and safe. This stability means resource adjustments like CPU and memory allocation are smooth and predictable, reducing the risk of performance spikes or resource starvation. It enables regular benchmarking and AB testing to ensure the RL agents decisions are actually improving the cluster perform. We track key metrics such as application response time, resource utilization, and cost efficiency before and after policy changes. Regular benchmarking and AB testing help us validate that the agent is learning effectively and not just over fitting to the recent data. Safe exploration is critical in production environments and our agent must try new actions to learn, but reckless experimentation can disrupt services. We use techniques like reward, shaping, circuit breakers and canary DI deployments to limit the impact of risky decisions. This ensures that learning continues without compromising reliability or user experience. Soft factor critic for multi objective optimization. SAC is a state of art, deep reinforcement learning algorithm designed for environments where actions are continuous. It is especially valued for its ability to balance exploration and exploitation, making it highly effective for complex real world optimization problems like Bernet, workload management, entropy, regularization ease, A technique used in SAC to encourage the RL agent to explore a wider range of actions rather than settling too quickly on a single strategy. In cloud environments, workloads and resource demands can change rapidly and unpredictability. Exploration helps the agent discover new, potentially better resource allocation strategies that static or overlay conservative approaches might miss by maintaining a balance between exploitation and exploration. SAC ensures the system remains adapt, adaptable, and can respond to novel situations. One of S's strength is its ability to handle multiple objectives simultaneously. In Kubernetes, we often need to optimize for cost, performance, and reliability All at once. SSE allows us to define a reward function that incorporates these different goals. So the agent learns to make decisions that doesn't, that don't just maximize one metric at the expense of others. This holistic approach leads to smarter resource management, where clusters run efficiently, applications stay responsive and cause TER control. In our experiments, we found that SAC was particularly effective in scenarios with highly variable workloads and complex resource requirements. For example, during a multi-service deployment with fluctuating traffic, SAC consistently achieved lower response times and better cost efficiency. Compared to DQN and PPO, its ability to adapt to continuous change and optimize. Across multiple objectives made it the best choice for our most demanding cloud native applications. Integration with Kubernetes controllers integration. Integrating reinforcement learning models with Kubernetes. Controllers present several challenges. One major issue is latency. RL agents needs timely data to make decisions, but delays in metric collection or a PA calls can impact performance. API Compatibility is another hurdle. Kubernetes APIs evolve and custom controllers must stay up to date to ensure smooth communication between the RL agent and the cluster. We also had to consider security and access controls, making sure RL agents only perform safe authorized actions within the cluster Custom Controllers act as translators between the RL models and Kubernetes. They colored cluster metrics, feed them to RL agent, and then convert the agent's recommended actions into Kubernetes. API calls this architecture allows us to leverage the intelligence of RL while maintaining compatibility with existing Kubernetes workflows. Controllers also handle error shaking and rollback procedures, ensuring that any action taken by the RL agent is safe and reversible. Monitoring oral driven actions is critical for both safety and transparency. We learned that deep integration with existing observability tools like promises and Grafana helps track the impact of oral decisions in real time. Debugging. Oral agents can be tricky, especially when decisions seem counter initiative. We found that logging details, state information and reward calculations were essential for diagnosing issues. Regular audits. And automated alerts for unusual behavior helped us catch problems early and maintain trust in the system. Let's discuss about some multi cluster environment challenges. Synchronizing state across multiple Kubernetes clusters is a complex challenge. Each cluster may be running in a different region or different hardware, or even in different cloud providers. Ensuring that all clusters have a consistent view of workloads, resource usage, and policies request. Robust coordination mechanisms. We found that network partitions, version mismatches and a synchronous updates can easily lead to inconsistencies. Making reliable synchronization a top priority. Latency is a critical factor in multi cluster environments. Decision about workload placement or scaling must account for the time it takes to communicate between the clusters. High latency can delay resource adjustments leading to suboptimal performance are even service disruptions. Our approach includes latency awareness, algorithms that factor in network delays, helping ensure timely and effective resource management across distributed clusters. Federated learning offers a promising solution for scaling reinforcement learning across clusters. Instead of sharing raw data, each cluster train its own oral agent locally and only shares model updates. This approach preserves data privacy and security as sensitive information never leaves the cluster. Federated learning also enables collaborative optimization, allowing clusters to benefit from shared insights while maintaining autonomy and compliance with data regulations. So implementation of framework with CNCF tools. CNCF tools. CLOUDNATIVE Computing Foundation tools are open source projects and technologies hosted by Cloud Native Compute Foundation, designed to support cloud native infrastructure, application development, and operations. These tools help organizations build, deploy, and manage scalable, resilient, and observable cloud native systems. I have listed below some common CNFC tools promises. It is essential for RL training because it provides a comprehensive and real time view of cluster metrics, CPU memory network uses, and customer application metrics. These rich data streams allow the RL agents to make informed decisions and continuously learn from the actual state of cluster. Without accurate and granular metrics, RL models will be flying blind, unable to adapt to changing workloads or optimize resource allocation effectively. Istio as a service mesh gives us fine grain control over traffic routing between microservices. By integrating RL with TO, we can dynamically adjust routing policies based on real-time performance feedback. Sending requests to the healthiest endpoints are balancing load to optimize latency and cost. This level of automation helps maintain high availability and responsiveness even as traffic patterns shift unexpectedly. Helm, it simplifies and standardizes the deployment of RL optimized workloads by packaging configurations into reusable charts. This reduces manual errors, speeds up rollouts, and ensures consistency across environments. Whether you are deploying to a single cluster or multiple region. With Herm updates and rollbacks are straightforward, making it easier to manage complex deployments and maintain reliability as you iterate on oral models. Let's talk about the performance benchmarking strategy. Benchmarking is essential to objectively measure the impact of R driven optimization. By comparing cluster performance before and after oral deployment, we can quantify improvements and identify areas for further tuning. It is important to establish a baseline using traditional auto-scaling methods so we have a clear reference point for evaluating our ALS effectiveness. Consistent benchmarking also helps ensure that changes are beneficial and don't introduce new issues. In our RL optimist clusters achieved a 30% reduction in application response time compared to static auto scaling. We also observed higher CP utilization and fewer unnecessary scaling evens. Which translated into better resource efficiency and lower cloud costs. These concrete results demonstrate the real world value of intelligent workload optimization. Benchmarking isn't just about numbers. It's about building trust. Stakeholders need to see clear, repeatable evidence that RL automation is safe and effective. Transparent reporting of performance metrics, REASSURES teams that the system is making smart decisions and not compromising reliability. Regular benchmarking and sharing results with the team. Foster confidence in adopting R driven solutions for critical workloads. Now let's consider some production deployment con considerations as we move from research and benchmarking to real world deployment. Safety and reliability become paramount. First, we implement safety mechanisms like circuit breakers, where these prevent RL agents from making destructive or risky decisions during training or model updates. If the agent starts to behave unexpectedly, the circuit breaker Hals its actions protecting the cluster from instability. Next, we use a gradual rollout strategy. Instead of deploying new RL policies cluster where we start with Canary, Dell deployments rolling out changes to a small subset of workloads. If performance degrades, automated rollback mechanisms quickly revert to the previous stable policy. Minimizing risk model versioning is also critical. We maintain a comprehensive versioning and AB testing framework allows us to compare the performance of different RL models side by side. This ensures that only the best performing models are promoted to production. Finally, observability integration is essential. We deeply integrate with existing monitoring tasks such as promises and Grafana to audit RL agent decisions and model performance in real time. This transparency built trust and allows for rapid troubleshooting if issue arises. In summary, by combining safety mechanisms, gradual rollout, robust versioning, and strong observability. We ensure that RL driven optimization is not only effective, but also safe and reliable for production, kubernetes and environments. Next steps. What are the next steps for implementing RL Optimiz? Optimiz Kubernetes? When adopting RL driven optimization, it's wise to begin with pilot projects targeting noncritical workloads, and this approach allows you to validate the benefits and iron out any issues before scaling up. Pilot project help teams gain hand-on experience with AL tools and workflows, building confidence in the technology. By starting small, you minimize risk and ensure that any expect unexpected challenges can be addressed without impacting core business operations. Before deploying AL solutions, it's crucial to establish baseline metrics for your current system, such as resource utilization, response times, and cost. These baselines provides a reference point for measuring improvements and help you set realistic goals for RL optimization. Tracking metrics before and after implementation, ensure that changes are data driven, and that you can clearly demonstrate the value of RL two stakeholders. The Kubernetes and RL communicates are vibrant and collaborative. Sharing your learnings, challenges and successes can accelerate progress for everyone contributing to open source a frameworks or Kubernetes tools. Not only help others, but also brings valuable feedback and innovation to your own projects. Community collaboration fosters best practices. Drives new features and helps build a more robust ecosystem for intelligent workload optimization. So what are the key takeaways from this to wrap up? RL Optimist Kubernetes delivers three major benefits. Efficiency resources are used more effectively, reducing waste and lowering costs. The second one is adaptability. The system learns and responds to changing workloads, keeping applications running smoothly, even during unexpected spikes or dips. Third, it's a future proofing rls. Continuous learning means your infrastructure can evolve with your business needs, new technologies, and shifting user demands. So successful adoption. Adoption of RL in Kubernetes isn't a one step process. It's best approached in phases, starting with pilot projects, establishing baseline metrics, and gradually expanding to production workloads. This phased strategy helps manage risk, build team expertise, and ensure that each step delivers measurable value before scaling up. Ultimately, our optimist, Kubernetes is more than just a technical upgrade. It's a strategic advantage. It empowers organization to be more agile, resilient, and cost effective in a rapidly changing digital landscape. By embracing intelligent automation, you position your infrastructure and your business for long-term success and innovation. With this, I conclude my presentation. Thank you.