Hello everyone, this is Peush K. Thank you for joining me today here I have more than 20 years of IT experience in US healthcare, developing and maintaining software projects and programs. So here today we are going to talk through a topic that's rapidly gaining traction across healthcare it implementing AI driven observability. As SE practices for reliable healthcare systems, the integration of artificial intelligence in healthcare systems offer tremendous potential to improve patients' outcomes, but it is also presenting unique reliability challenges when AI systems supporting critical healthcare functions experience downtime. The consequences can directly impact patient care. This representation and this presentation, which we are going to discuss here will say how site reliability engineering SRE practices are being successfully adopted to maintain exceptional availability in AI powered healthcare platforms based on real world implications. Across multiple healthcare institutions, our focus on how these practices ensure the reliability of AI powered platform is critical for healthcare environments, when even milliseconds of downtime can have a life altering consequences. Whether you are a healthcare provider, a tech vendor, or an SRE team. Is stepping into regulated domains, you will walk away with actionable insights and established blueprint for implementing these practices in real world environment. In this slide, we will discuss the AI healthcare reliability challenges. So we start how this unique reliability challenges. Are affecting healthcare, artificial intelligence, AI systems increasingly influence clinical decision making, diagnosis, treatment planning, and patient monitoring. A model failure or downtime doesn't just create technical issues. It directly affects patient's outcomes, regulatory compliance. Hipaa, FDA and other healthcare standards, we have to comply with all these standards based on the geography to successfully implementing and using AI in healthcare facility. From a regulatory standpoint, we face a stringent requirements, hipaa, which is health insurance Portability and Accountability Act in US GDPR. Which is General Data Protection Regulation in Europe and Food and Drug Administration, FDA, they all see AI as a medical device. This imposes a strict controls on how we update, monitor, and validate these systems. Healthcare AI system faces unique challenges that the traditional SRE approaches must adopt. To address patient safety directly depends on systems reliability, creating zero tolerance situation for certain types of failure. Meanwhile, complex regulatory requirements add additional constraint. To how system can be monitored, tested, and updated. Finally, there is a constant push and pull between innovation and reliability. AI evolves rapidly, but healthcare infrastructure is built on safety and stability. This mismatch requires. A refined approach to SRE one that merges technical robustness with clinical caution. So here, let's move on to third ipo. Important aspect, how it is affecting the case studies, which shows the automated observability and its benefits. The observability transforms into impacts our case studies, access multiple hospitals and shows a 78% reduction in meantime to resolution and TTR for critical incidents. But how? By integrating AI powered observability. Platforms that continuously monitor not just technical metrics, but clinical relevance too. These systems achieved a 92% variance detection rate, capturing early sign of AI drift or model flatness before they could affect care. 65% decrease in incidents we saw year over year in clinical incidents, and that is under 30. It is a practice. Our multi-institutional cases studies demonstrates dramatic improvements through implementing automated observability tools, healthcare organizations. Participating in our research deployed specialized monitoring solutions designed to detect subtle anomalies in AI prediction patterns before they manifested as clinically relevant issues. Organizations like Mount Sinai and Mayo Clinic are developing AI to prevent issues. Rather than reacting to them, the key is a shift from reactive monitoring to proactive variance detection using machine learning techniques on logs, metrics, and user behavior. Now, let's cover how establishing meaningful SLIs and SLOs. For healthcare, AI service level indicators, SLIs and service level objectives, SLOs shapes the foundation of SRE. However, in healthcare, they require a tailored lens. We categorize them into technical, clinical, and operational indicators. Let's cover technical sli. Inference latency, less than 200 milliseconds and model drift, which we aim to keep below 0.5% to ensure diagnostic relevance. Batch processing completion, 99.9% API Availabilities for supporting systems. 99.95%. Clinically, we track diagnostic suggestions, accuracy and override frequency, an indicator of clinicians trust in the system. Operational metrics includes time to detect variance. And recovery time objectives under clinical and uh, sli. You can see here that there is a documentation completeness score. There is a clinician's override frequency under operational recover, incident response time, recovery time objectives, and change failure rate. By aligning these with key clinical performance indicators, KPIs. Like treatment relevance or documentation. Completedness, we ensure that our AI goals serve both system and patients. The collaboration between SRE teams and clinician is very crucial here. SRE teams across participating healthcare organizations established novel SLIs and SLOs. Is specifically designed for AI systems. The most successful implementations balanced traditional reliability metrics with healthcare specific indicators tied directly to clinical outcomes. In one project, we found that tracking clinician override frequency was more informing. Then raw model accuracy, and it directly indicated trust and ultimate adoption. These metrics were developed through close collaboration between SRE teams and medical professional. Ensuring then technical reliability is aligned with the patient care priorities. Chaos engineering. In healthcare ai, a safe approach. When you see, and you read and listen, this word chaos, you definitely want to go far, far away if you are in the healthcare industry. But here I would say that the chaos engineering may sound risky in a healthcare context, but with the right safeguards, it's a game changer We conduct. Fault injection experiments in sandbox environment using synthetic patient data. That would be a isolated environment. Then we move on to control failure. As you see here, it is graduated fault injection with safeguards. What does it mean? So this allows us that we will go to conduct intentional. Fault injection experiments in the Sandboxed environment. This allows us to test resilience of AI system. Like what happens if an inference service slows down or if a prediction pipeline fails. Every step is reviewed by clinical experts and before it even go to pre-production, and the result is. Better system is strengthening. For example, we identified a cascading failure risk in a sepsis decision model that would have gone unnoticed without simulated faults. With a graduated testing approach and a tight clinical oversight, chaos engineering becomes not just safe. It essential for reliability. Our framework for chaos engineering practices safely applied to healthcare AI systems and it revealed how controlled failure testing identified resilience gap in prediction pipelines before they affect production environment. Organization implemented, graduated approach. Beginning with fully isolated environment using synthetic data as confidence grew more sophisticated. Failure models were tested, and always with multiple layers of safeguards and clinical experts oversight to prevent any potential impact to patient care. Now let's cover the error budgeting. For clinical AI implications, error budgeting is another core SRE principle, and it adapts beautifully to healthcare ai. Here we define reliability targets based on the clinical impacts of each system component. For instance, an AI system generating post visit summaries. Has lower risk compared to one recommending oncology treatment. We are going in a step by step progression here. Define reliability targets, then allocate error budgets, then move on to implement graduated alerting, which is a tiered by budget consumption rate. Then we move to balance innovation and stability. Where we adjust deployment pace based on the budget of the facility, we distribute error budgets accordingly. We use safeguard alerting and aligned deployment pace with the remaining budget one. Hospitalist studied. We saw a 64% reduction in non-actionable alerts after implementing this approach. How impressive. Importantly. This allowed them to innovate more rapidly in low risk areas while safeguarding critical functions. This tiered approach allows more aggressive innovations in the low risk areas, and enforcing is stricter controls or components directly influencing clinical decisions. And as I just told you, it's very impressive reduction rate. So now let's discuss how continuous verification of AI model in production is required. We all know that AI is based on the data and we, nobody can disagree that the real world data is ever changing. So unlike traditional softwares. AI models degrade over time. So what happens here? Continuous verification helps us manage that degradation. Predeployment, we validate models against benchmark dataset. We then use shadow deployments running new models. In parallel without affecting live outcomes to observe their performance. AI models are living systems, they degrade, and so we treat model verification as a continuous multi-phase pipeline. The step we took here is all you can see on this screen, pre-deployment validation, shadow deployment in parallel. Canary rollout with rollback triggers and live drift monitoring and age case flagging. Canary release allow controlled traffic shifts with rollback triggers. One organization reported a 72% drop in model related incidents using this multi-stage verification. Continuous verification techniques in AI model enable safe deployment. And this approach combined traditional AB testing with healthcare specific verification steps, including clinical review panels, and, and we have seen an impressive reduction in the incidents Importantly. We don't just rely on statistics. We include clinician panels to review age case scenarios before full rollout. This ensures safety while supporting innovation. Next slide. We are coming here. Git Ops for healthcare, AI compliance and reliability. I have captured into three broad segments here, infrastructure and code, automated deployment pipelines, and then compliance automation. Git Ops brings the consistency of software engineering into healthcare AI infrastructure. We define infrastructure as code and what does that mean? We versioned infrastructure definitions, declarative system configurations, automated compliance validation, and change history documentation. We have continuously watching the continuous integration and continuous deployment ci cd pipelines here. This supports regulatory mandates from hipaa. High Trust, which is health Information Trust alliances and GDPR by ensuring DUPLICABILITY and auditability. Automated pipelines include clinical sign-off and auto rollback mechanism. Automated deployment pipelines also covers that SLO value violations. We would be able to roll back automatically. Immutable artifact management, progressive delivery patterns, PHI, which is protected. Health information is tightly controlled and rollback is triggered if the SLO violation is detected. Git ops let us deploy changes confidently. Even in regulated environments. Several hospitals, networks are now using tools. Like Argo CD and Flex CD for the same purpose. This is how we are really using and enhancing the SRE system. Let's talk about the financial impact of SRE in healthcare ai. So that's all the numbers we are showing here. Initial investment in observability and SRE tooling is non-trivial. Definitely there is a initial, significant investment is required, but the ROI comes fast across cases studies. The average time to break even was seven and a half month. Some institution got returns in as little as four months. Financial analysis across participating organizations demonstrates that despite initial investment healthcare system achieve substantial returns within months, there is also a softer but critical benefit. IT clinicians trust and satisfaction improved due to fewer disruptions. Leading to higher adoption rate of AI tools, one large hospital system saved over $1.2 million annually. After adopting these practices and saw a noticeable boost in AI adoption and clinical satisfaction, how impressing it is because if the clinicians are satisfied. They will be motivated to use the AI tools, and that is where we need to make sure that their satisfaction index continuously grows and improves through SRE practices. Here comes the time when we want to bring all of that together. If you are thinking of adopting these practices, here is the blueprint. First of all, we need to make sure. That we formulate a cross-functional team and what the team comprises of it should have SRE engineers with healthcare background, clinical stakeholders, compliance and security specialist, and then AI ML engineers. Then we move on to assessment and planning. Then these assessments maps out. Critical system and evaluate your technical depth. It is part of system inventory, current reliability metrics, baseline compliance, requirement mapping, and technical depth evaluation. Third step would be foundational implementation, including observability tools. Service level objectives, their definitions and continuous implementation, continuous deployment, pipeline improvements. It includes the, and that's the foundational, because we don't want to go at a larger scale in the beginning, and that's very important aspect here. So we go slow, but we go. With less and less errors and more benefits, uh, very soon, organizations that follows these phased approach showed significantly higher reliability scores and smoother scaling of their AI initiatives. This actionable blueprint provides a structured approach for SRE leaders. Implementing and maintaining reliable AI systems. Our research identified that organization that achieving the highest reliability score followed a similar implementation pattern. Starting from the cross-functional team formulation and to ensure both technical and clinical perspectives are well represented. The most successful implementation maintained a phased approach. Establishing core observability and measurement capabilities before advancing to more sophisticated practices like chaos, engineering and advanced automation. So what are the key takeaways and next steps? So, healthcare, ai, right? It needs to be specialized AI approaches. A standard SRE practice must be adopted to address the unique reliability requirements and regulatory constraints of healthcare environment. Financial ROI is achieved and it's significant, but before we come to the ROI, we also want to make sure that we go with the cross-functional corroboration and then we go smart small. Measurable constantly and then expand method methodologically. So the most successful implementation bridge the gap between the technical reliability and clinical impacts through structured collaboration between SRE teams and healthcare professionals. The future of AI in healthcare. Depends on our ability to make it safe, scalable, and stable. Despite initial investment requirement, healthcare organizations constantly achieved substantial returns through improved reliability, reduced incidents, and better resource utilization. The integration of AI in healthcare. Continues to accelerate making reliable operations, not just a technical consideration, but patient safety imperatives. By implementing the SRE practices outlined in this presentation, healthcare organizations can reconcile the seemingly competing demands of rapid AI innovation. With the stability requirement of critical care infrastructures, as I told you that SRE is the crucial part of this journey. Thank you.