Conf42 Platform Engineering 2025 - Online

- premiere 5PM GMT

Platform Engineering at Scale: Building Cloud-Native Data Infrastructure for 140TB Monthly Healthcare Analytics

Video size:

Abstract

From 12-hour queries to 2.4 hours. From 15TB to 140TB monthly. $2.3M saved. See how platform engineering transformed a Fortune 500’s healthcare data chaos into a self-service powerhouse that accelerated drug discovery by 4.2 months per compound. Real metrics, real impact.

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello everyone. I'm Kumar and I'm excited to share my insights gain from two decades of hands-on experience in technology and engineering. Throughout my 20 year journey, I've had the privilege of working on diverse transf transformation projects that has shaped my understanding of platform engineering. At scale. Today, I will share one of our most impactful implementations. Healthcare analytics and that is platform engineering. At scale building cloud native data infrastructure for one 40 terabyte of data monthly. Healthcare analytics. Healthcare organizations face unique data engineering challenges. Managing hundreds of terabytes of sensitive data monthly while maintaining HIPAA and GDPR compliance create significant complexity combined with legacy system constraints, demands for self-service analytics, and the need to integrate diverse data domains. These challenges directly impact medical innovation and patient care. Modern platform engineering solutions have become essential to address these critical barriers. As a Fortune 500 biopharma company, legacy infrastructure, severely constrained research capabilities, the platform access only 15% of available healthcare data sources with the researchers facing two hour query rate times. The monolithic system couldn't scale to meet growing data demands effectively limiting innovation and delay potential medical break breakthroughs. Our platform engineering transformation, increased month monthly processing capacity from 15 terabyte to one 40 terabyte reducing processing time by 80% and accelerating drug discovery timelines by 4.2 months per. We developed a platform as a product approach, treating infrastructure as a product with research and clinical teams as customers. By implementing self-service capabilities and automation, we achieved 67% faster development velocity and reduced processing time by 80%. The platform integrated compliance. Requirement directly into infrastructure code while enabling team to access resource without engineering bottlenecks. This transformation delivered $2.3 million in annual cost savings and increased workload capacity, demonstrating the value of treating platforms as products rather than just infrastructure. Our healthcare data platform leverages a W Services to process our one 40 terabyte of monthly data efficiently. Through automated e retail pipelines, multi-zone data architecture, and dynamic scaling capabilities, we achieved an 80% reduction in processing time, the platforms metadata driven framework, and choose data quality while maintaining HIPAA compliance. Through built-in security controls, this architecture uses S3 Amazon Redshift to store to access the data and Amazon S3 to store the data. There is also addition layer in between, which is nothing but the copy command you can use to load data directly into the Amazon Redshift. Our INFRAS infrastructure as code approach integrates compliance and automation directly into our development deployment pipelines. Through automated infrastructure provisioning and self-service capabilities we have achieved faster developer velocity and seamless scaling from 15 terabyte to one 40 terabyte monthly capacity. Platform's compliance as code approach and embeds regulatory requirements infrastructure, while automated scaling and optimization have delivered major cost, annual cost savings. This transformation reduce deployment times while maintaining consistent security controls across environments. Our platform engineering initiatives delivered significant outcomes across key metrics. We achieved an 80% reduction in processing time through automation, increased workload capacity by three 40%, and realized $2.3 million in annual cost savings. Developer velocity improved by 67% through self service capabilities. Monthly processing capacity scaled from 15 terabyte to one 40 terabyte. These results demonstrate the substantial impact of our platform transformation. Our platform transformation delivered significant healthcare innovation outcomes. Drug discovery timelines accelerated by 4.2 months per compound per month. Through faster data processing, clinical research accuracy improved by 45 40 5% through enhanced data integration. While clinical trial success rate increased by 23% through improved data accessibility, these results demonstrate tangible impact on healthcare innovation and research efficiency. And our platform engineering approach creates infrastructure as a code with research and clinical team as customers. The mindset shift transformed our delivery model and team interaction, enabling healthcare teams to focus on innovation rather than infrastructure. Our legacy to cloud near transformation followed five key phases. Assessment, evaluated capabilities and defined target architecture, foundation, established cloud, environment and compliance frameworks. Migration. Executed workload transition using blue, green, deployments, optimization, refined architecture, and enhanced cell service capabilities. Innovation, expanded platform capabilities with advanced analytics and ML integration. This phased approach enables seamless transformation. While maintaining operational continuity, we face key challenges, including organizational resistance, skill gaps, legacy integration requirements, compliance concerns. Balancing standard ideation with flexibility solutions included executive champions driving cultural change, embedded platform engineers phased migration with clear metrics, compliance as code implementation and golden parts with escape patches for edge cases. This comprehensive approach address both technical and organizational challenges. The implementation focused on four practical strategies. Start small and scale fast with defined high value use cases. Embed compliance into platform automation and CICD pipelines. Measure platform value through clear KPIs, including developer velocity and resource utilization, and build internal community through feedback forums and train training programs. These approaches grow successful platform adoption and continuously success in healthcare. Platform engineering requires four key elements, product, mindset, self-service, acceleration, automated compliance, and meaningful metrics. Tracking these principles enable our platform to deliver 80% faster processing and accelerated healthcare innovation through improved. Clinical research accuracy and drug discovery timelines.
...

Ritesh Kumar Sinha

Associate Vice President - Relationship Management @ Kotak Mahindra Bank

Ritesh Kumar Sinha's LinkedIn account



Join the community!

Learn for free, join the best tech learning community

Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Access to all content