Conf42 DevOps 2023 - Online

Humana DevOps Journey - The Road Less Traveled

Video size:

Abstract

We, at Humana, Fortune 41 company, started our DevOps journey in 2018, when organizational change lead the way and we are excited to come back to provide an insightful update on our current DevOps journey and how our people are driving the change today, to keep the momentum going and serving our Humana Engineers, by providing the tools, platforms and streamlined processes so that they can continue to create delightful experiences for our Medicare, Medicaid, Tricare and Pharmacy customers. We are working hard each day to challenge the routine and pioneer simplicity and build a data-driven mindset around change, letting people drive the change based on data. In our session, we will provide you with some key lessons learned around how growing people and their engineering practices actually creates the differentiator for a business and not the technology or platforms that an organization purchases.

Summary

  • Rupesh Kumar is the principal DevOps working with Andrew Foster who is the director of DevOps Engineering at Humana. And we would like to share some insights which you can take it back to your organization.
  • Andrew Foster is the director of DevOps Engineering at Humana. His team is responsible for all of the DevOps platforms. For Humana, we care for the engineer experience as well as the engineers tooling. And I love it every day.
  • Humana is a Fortune 41 health and wellness company. Our primary business is serving our 5.1 million individual and group medicare advantage members. How does DevOps fit into that? Our mission is to give engineers the most compelling development experience possible.
  • Our DevOps journey started at Humana in 2018 with the leadership team wanting to drive change. Culture is what makes people understand each other better. Humana's strength is in developing a uniqueness and to always question the status quo. A happy employee makes the happy customer.
  • In order to make things happen incrementally and iteratively, it's very important for us to kind of like listen to the customers. Next thing is empower. The team is, I mean, it takes a village to drive a change. Andrew Rapesh will now talk about how we are scaling DevOps for a large organization like Humana.
  • There are really three key steps and three key kind of methodologies that unlocked our ability to really scale DevOps. The first one is alignment and feedback. The second key was to focus on platforms and tools that are engineer focused. And then the final bit is smaller releases, right.
  • Greenlight API is our patented solution to incorporating Humana's SDLC into the DevOps pipelines. It gives the developers feedback as they go upon their software development journey. And it's one of the things that has really helped us unlock our DevOps journey.
  • We consider open systems, which would be our traditional net, Java, Python, et cetera, programming languages. And then of course, mainframe has its own experience. We really are pushing towards a sense of ownership for development teams to build and own their pipelines.
  • Humana has over 1600 applications and components with over 300 teams. We have over 10 million lines of code with over 17,000 CI CD pipelines. Over 56% of automation releases over manual releases. 97% of our releases are eligible for releases in production.
  • Rapesh: As we all know, DevOps is certainly a journey and not a destination. Hopefully next year Andrew and I will come back again to share new challenges and opportunities. With that, I would say thank you for tuning in and joining our session.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Morning everyone. My name is Rupesh Kumar. I'm the principal DevOps working with Andrew Foster who is the director of DevOps Engineering at Humana. And we are very excited here to bring forth our journey of DevOps and how we have done it. And we would like to share some insights which you can take it back to your organization and hopefully you will find some good value in it. So I have been working in this field for over 20 years and I've been working with both the federal sector and the commercial sector. And as of recently we have been doing some good, amazing work in DevOps outside the industry. Now Humana has leading that effort and Andrew Foster certainly has been leading this from past few years. And I have joined his team and we are kind of making DevOps at a scale. So that's where I come in and I would hand it over to Andrew to kind of introduce and take it away from there. Thank you, Rapesh. Great intro sir. Welcome everyone to our session again. My name is Andrew Foster, I'm the director of DevOps Engineering here at Humana and my team is responsible for all of the DevOps platforms, whether it's open systems, mainframe or salesforce. For Humana, we care for the engineer experience as well as the engineers tooling. So everything around engineering software. We are the it team for the it teams in which we take great passion in. I've been at Humana since 2011 holding various technical roles and been doing the DevOps for the past four years. And I love it every day. Rapesh, could you hit the next slide, sir? Sure. Thank you. Before we get into much about our journey, as Rupesh mentioned earlier, we wanted to tell you a little bit about Humana. So Humana is a Fortune 41 health and wellness company. We view ourselves as more than just traditional insurance and we are what we call a human care company. We take a unique approach to healthcare that's completely centered on people and all the things that they need to feel whole for just a little bit of context. We're the fifth largest health insurer in the United, started among publicly traveled insurers, and our primary business is serving our 5.1 million individual and group medicare advantage members. We also serve our active duty and veteran military families, offer group insurance and other standalone wellness products. All right Rapesh, hit the next slide, sir. And so as you can tell at Humana, we really value our members and everything that we do. All of the software that we deliver is in support of our members to make their experience better while they're using our products and using their healthcare benefits. So how does DevOps fit into that? Well, our mission is to, as you see on the slide, give the engineers the most compelling development experience possible, and we take that seriously. We aspire to get out of the engineers ways, allow them to focus on software delivery only, and give them the best experience possible while they're doing that. Repesh, I'll hand it to you to get into how our Humana values have inspired the DevOps transformation here at Humana. Thank you, Andrew. So, as Andrew said, our DevOps journey started at Humana in 2018 with the leadership team wanting to actually drive change. Now, as you can see, our core values define our culture. Now, culture is what makes people understand each other better. You may be wondering why culture has to do with DevOps. Well, we say that culture is pretty much everything to do with DevOps, because everything is driven by the people, for the people, and of the people. So that's where DevOps actually fits in very perfectly. Now, our strength is in developing a uniqueness and to kind of always question the status quo. And that's pretty much what we always do as part of this team. We have a diverse workforce, and we are always able to bring in their unique strengths and even the weaknesses and kind of see how we can complement each other. Now, with the different thinking styles of people we have in our workforce, it always good to look at the problems from different perspective. And that has always helped us to actually look at the status quo and keep innovating, if you will. Now, as Leonardo da Vinci once said, I remember that very clearly in one of the book which I read. It said, simplicity is the ultimate sophistication. So one of the things which we do at Humana with respect to DevOps is to make things simple, as Andy was saying earlier, because we do it for our developer community and at the end of the day, for the customers. So we need to make the process simple, which can be understood and kind of like easily scalable as well, at the same time. Now, one of the other things which we also do at Humana is focus on health. And certainly, needless to say, because we are the healthcare company, so health is one of the utmost things. So we inspire that the person is always emotionally, physically, and spiritually available, and they are inspired to work in a community. And in order to do that, we actually have lot of different online communities, channels, talk shows, where we talk about the overall health of the individual, because a happy employee makes the happy customer. And that's what at least I believe in. So that's where I would say our company culture has trickled down into our DevOps, and that's what we see every day, and we live and breathe that. So with that said, I will actually take you on to the next screen where we talk about how we have developed this Humana DevOps model and which has been working for us pretty efficiently. Now, in order to make things happen incrementally and iteratively, it's very important for us to kind of like listen to the customers. So one of the things which we have started doing is we are paying more attention to what the customer needs and wants are by doing active listening. Most of the time you may feel, hey, that something is a second nature. We always do that. But you will be surprised how much we can learn when we are actively listening to our customer. And even for things where they are not even saying it, but they may actually be talking around it. But then the good thing is to listen to the customer, pay attention to what they are saying, and kind of identify where their pain points are. And that actually has helped us tremendously to build on some of the value which we introduced, which created a big win for us, or some may call it as a low hanging fruit. So certainly active listening is certainly the big one. And then the other one is kind of making sure that you are repeating yourself to the customer to understand that you have understood it right, because that's something which I have seen many a times where we miss on the expectations of what customer really is saying versus what we are kind of understanding. Next thing is empower. Now, how do we make sure that the customer is providing us the requirements and what he wants to get done and who is making it happen? So we certainly believe in that. The team is, I mean, it takes a village to drive a change. So the same thing goes in our DevOps. So we have a culmination of diverse team members who come from different backgrounds, and we empower them to actually innovate and even to fail smart fail rather quickly than later. So that's where we provide all the capabilities to the developers to come in on the platforms to actually talk and experiences and see how things can happen. So empower is one of the key things which we do, and then certainly the guide. So making sure that when you are bringing on DevOps to a large organization like Humana, you have to have a set of defined principles, standards, patterns, policies, because it could very well go in many different directions. And certainly we want to make sure that there is a standardized process for doing DevOps and in order to do that, one of the things which we have started working on is bringing principles, standards, policies around, how to do the branching strategy, how to do progressive delivery, how to do Gitops, how to do environment strategies and microservices. So quite a few things which are out there. We have standardized it so that the developers can feel that they are part of the community and they can work together in not only digesting it, but also helping out other development communities, which we already have in Humana. So we have a team of over, we have over 400 plus teams within Humana. And I think it takes a village, again, to kind of make things happen. And that's where this guide certainly helps in doing that. The other important thing as part of this model is the data. How do you collect the data? Like making sure that in order for you to know that is my DevOps improving the user experience, is it helping the developer do the right way of doing DevOps? We call it as realistic DevOps. Now, in order to gather that metrics, we certainly have to make sure that everything is flowing into data. Lakes and tools like big data superset comes to rescue, where we can mine the data, we can analyze the data, we can run simulations on it and see how it goes. And we build our okrs, the objective ski and results around it as to how we can iteratively and incrementally improve the overall experience of DevOps for the developer community and eventually for the customers as well. So gather the data and then certainly measure the data depending upon different tools which you have and kind of make that happen. So I guess with that, I would say that currently this is our model. I would not say that this model is the only model, but this is the model which currently works for us and we are using this every day to make it better for the life of the developer. And Andrew will now talk about what we have done so far and the other aspects of how we are scaling DevOps for a large organization like Humana. And then he will also share some of the key personas which exist in Humana. And we'll wrap it up by talking about some of the benefits and some of the return on investment which we have seen with DevOps. With that, I will hand it over to Andrew. Thank you. Rapesh. Great job summarizing our approach. I think the emphasis on the engineer and how we can get feedback and support that experience, that's really the key. So let me get into a little bit of specifics right about how do we do the DevOps in a large enterprise like Humana. And I'll take you guys all the way back to 2018 and kind of the approach that we took as we move forward, there are really three key steps and three key kind of methodologies, if you will, that unlocked our ability to really scale DevOps. The first one I will talk about is alignment and feedback. It's hard to really do anything in a large enterprise, as many of you know, I'm sure many of the folks watching here work in large corporations without alignment and a go forward strategy. And I would say in 2018, Humana was in a unique position, organizationally, where we achieved a significant amount of alignment across a broad spectrum of the engineers community, which allowed us to really build momentum and take significant ground towards DevOps adoption. So that alignment across the organization, across every segment, delivering software, that is probably one of the key, most foundational things that has unlocked everything that we've done as we've moved forward. The other piece of it is once you get that alignment, it's really tight feedback loops. So regular updates to leaders, regular feedback sessions with engineers to make sure that we are, in fact, building and delivering the capabilities that they, in fact, need. So those regular, really tight feedback loops helped us stay on track and continue to stay in that, I would say, state of alignment across the organization. The second key was to focus on platforms and tools that are engineer focused. And the key to that was we kind of set a rule at the beginning, is that in order to release software to production, we don't want an engineer to have to leave their native platforms. So our goal was to have our Humana engineers stay within the DevOps platforms and manage their change through our change process using their pipelines. And for them to never have to leave that interface, they should never have to go to a CMDB tool, to some kind of change management tool. They stay within their native tooling and get all the feedback they need, and the processes are integrated there at that point. So that was really key. And then the other thing is just platforms that are engineer focused, right? Platforms that help engineers understand what their posture of their software is that they're creating, whether it's secure or whether it's quality software. These things need to be integrated into their workflow. And just not only that, but just the core platforms should be easy to use. Right. And engineer first. And we not only did that for our open systems, but we also took an approach where we have a unique experience for our mainframe engineers. Humana still has quite a bit of mainframe activity or development, as well as Salesforce. So we have a unique set of tooling for each of these personas. And then the third step, I would say that was most important. And I'll talk about this a little more detail on the next slide. But Humana is a highly regulated, we're in a highly regulated industry within healthcare and primarily our Medicare business, working with the federal government. And so it's really important to us from a business perspective that we adhere to change best practices and that we have traceability and that we're passing our audit and compliances and not only to our business, but it's important to our members, right, that we're operating our software in a secure way and that we're running our business in a secure way from a software perspective. So we were able to come up with a very unique, actually now patented solution to automate Humana's SDLC. Not yet, Rapesh. Go back to the other side, if you don't mind, sir. You're fine. Jumping ahead. You get excited. I love it. But we took a very unique approach to automating those checks into the pipelines. Again, keeping that approach that we want the engineers to stay within their native tooling. And then I would say the final bit is smaller releases, right. In order to do DevOps, we have to release smaller in order to iterate and move faster. And that's actually safer and healthier for a software development team. And so driving our releases and creating a strong mapping between application components and repositories not only allowed us to categorize our source code in a unique way, but also to help have an enterprise push to drive towards those smaller componentized releases. So I would say those three key things really, and there's a lot in that, right? But those three high level things really helped Humana unlock DevOps. And we've been on a really fast ride ever since we've done that. So Rapesh with that, if you don't mind. Jump to the next slide, sir. Thank you. So let's talk a little bit about Greenlight API. So this is our patented solution to incorporating Humana's SDLC into the DevOps pipelines. And so what you see around kind of the traditional DevOps Infinity sign that we've all seen, you have the Humana SDLC, right? The phases of that. So plan, deploy, sorry, plan, build, deploy, accept, feedback. You have the phases of that worked in to the pipelines in such a way that Greenland API, which is again, kind of our homegrown Humana naming, but Greenland API is a task that lives in the DevOps pipelines, it's required and in fact enforced to be there. And it gives the developers feedback as they go upon their software development journey. So as soon as they make a commit into their repo and they run their build pipeline, Greenlight API is telling them which parts of the SDLC they've adhered to or which parts they haven't. Not only that, but it's enforcing not just SDLC, but also things like security scanning, code quality scanning, a tight relationship between the source code repository and the change management or CMDB database. These things are really important in the way that we stitch the releases together for the engineers and stitch that feedback together, right? So that by the time the engineer gets to the point where they're ready to release to production, they've adhered to the Humana SDLC and all the requirements that Humana has to deploy software just by using their pipeline, right? And then an integration with our change system, which is in service now, as you see in the diagram, automatically opens change orders for those developers and so sends them through the approval process. So literally, DevOps do not have to leave their pipeline interface to release software in a certified and trusted way at Humana. And it's really helped us to get to a repeatable process where again, engineers are first, right? Because while yes, we're a large enterprise and we have rules and regulations and things that we have to follow to our jobs, that doesn't mean that we can't do it in an engineer first way. And so that's what we've done here with greenlight API. So we're very excited about this, very proud of it. And it's one of the things that has, as you saw in the last slide, that has really helped us unlock our DevOps journey. So let's talk a little bit about our developer experience. Again, I mentioned earlier, we have a unique diagram and kind of experience for each of our personas. So we consider open systems, which would be our traditional net, Java, Python, et cetera, programming languages, to be open systems. And then of course, mainframe has its own experience. And Salesforce, we won't get into those as much today. We've actually done other talks on our mainframe dev experience at BMC exchange last year. You guys can go find that presentation if you're interested in it. But for today, we'll focus on open systems. So again, you can see that we have the tooling aligned to the phases of our SDLC and which tool is used, at which point I won't just drain this slide right I think the audience can get a general overview and kind of understanding, especially since you're all DevOps practices. I think the most important thing I would call out here though is that all of this tooling is integrated, right? It's integrated in such a way that developers can self service, right? So if the platform didn't offer it, we've built automations in the thing that we call our developer control plane where engineers can provision their sonar queue projects or set up their artifactory JFrog repos. Right, to manage open source software. So we've taken great pain to integrate these solutions in a way that are easy for developers to use. But we're also, I would say, really passionate about a sense of ownership. So while as an enterprise team we offer these capabilities, we integrate them, we make them available and we provide tremendous guidance on when and how they should be used. We really are pushing towards a sense of ownership for development teams to build and own their pipelines because they are part of the application itself. And again, that's where our guardrails come in, right? That's where we have the governance of Greenlight API to ensure that while teams have ownership, they also have some oversight. Right, and that Greenlight API is going to ensure that before a release makes it to production, before that release is allowed to run and deploy to the production environment. All the checks have been followed, right? All the scanning happened in the build phase the right way. By the time in the accept phase we ensure that all the testing was completed in the right way, whether it was performance testing or manual testing. And then once they get to deploy, we make sure that the change management process was followed using Servicenow. And again, all that's automated. But there's some governance there to make sure that while teams have that ownership, they're still empowered to do it themselves. So very excited about this. It's been kind of the culmination of several years. So Rupesh, I'll kick it to you man, and let you talk a little bit about some of the value that we've realized from the work that we've done. Sorry Andrew. Thank you. I think I might have hit the slide. You were ready to quit too early, rapesh. We're not done yet. I guess this is where everything comes to the fruition of all the hard work, Andrew, you and your team, and done from past four years, I would say so, I guess. How did we do? So to put things in perspective, Humana has over 1600 applications and components with over 300 teams, which are members from around the world. Who are actually working day in and day out, except for the weekends and evenings, certainly, but certainly across different time zones. And we have over 10 million lines of code with over 17,000 CI CD pipelines. And these are pretty much managed and powered by over 24 tools which are owned by the DevOps platform team now. So I guess one of the high level statistics which I want you to take away from here is that we have seen over 56% of automation releases over manual releases. And when Andy was talking about the glappy, the green light API, which is one of the SGLC policy check kind of a tool which has been patented by this team. So this actually has allowed us to automate the releases to production, where it builds the trust with the change management and enablement team to actually make the releases go much quicker, much faster and with much quality. So 56% automation we have done. And there is an 80% of a lead time which has been experienced with respect to the dura metrics, certainly with the lead time, the wait time, the cycle time, the process time. And certainly this is the figures which we have run a couple of weeks ago. I'm sure this might be going up as we speak, because most of the teams are actually on the DevOps journey. And the teams which have taken on the DevOps adoption now, they are actually trying to mature DevOps and they want to do the right thing. So certainly you will see a bit more numbers going up as we speak. And then the overall we have, 97% of our releases are eligible for releases in production. What it basically means is that the entire CI CD stages of the pipeline, all the infinity loop which Andrew was showing earlier, all the different stages of the pipeline, they all meet the SDLC Humana standards. And that is a pretty big number for us. And I hope that in the future we keep exceeding and keep adding more metrics to capture more streamlined processes and kind of make it happen for the team. Now with that, I would say that certainly, Andrew, let me just take it back. Do you have anything else you would like to add here, Andrew, before we wrap it up? No, you did a great job. Rapesh. Thank you, sir. Thank you, Andrew. So I guess, as we all know, DevOps is certainly a journey and not a destination. So hopefully next year Andrew and I will come back again to share new challenges and opportunities which we came across and we overcame them in Humana. And we would like to kind of share our stories and certainly learn from your experience and stories as well. So with that, I would say thank you for tuning in and joining our session and hope you have benefited with our key insights and experiences which we laid. And hope you have a wonderful evening and other sessions with that. Thank you so much, Andrew. Thank you, everyone. Thank you. Bye, everyone. Take care,
...

Rupesh Kumar

Principal DevOps @ Humana, Inc.

Rupesh Kumar's LinkedIn account

Andrew Foster

Director - DevOps Engineering @ Humana, Inc.

Andrew Foster's LinkedIn account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways