This transcript was autogenerated. To make changes, submit a PR.
I'm Vilas. I am a director of engineering at Walmart.
I focus on enabling developers to deploy their code
in a resilient, performant way into the
cloud of their choice. That's what I focus on. What I'm going to talk about
today is a little bit about what we saw in our journey
towards becoming more resilient as a software
company and what were the challenges we faced, what are the mistakes
we made and all of that stuff. So before I go there,
obviously everyone in the room knows what this is.
I'm not going to repeat, everyone else is going to talk about this.
Suffice to say, the way we looked at this was we realized the importance
of this at Walmart, but we also realized that to
execute something like this at Walmart scale was going to be
a huge challenge. Right. What is Walmart scale?
These are some numbers. We have more than 11,000
stores worldwide that get supply of software
that actually does everything in the store. All of the management on
the supply chaos side, all of the retail business is managed by software.
The annual revenue of all of this combined is more
than half a trillion, which explains to the amount of goods that
are moved across the world for it. Right. That number,
270,000,000, is the number of folks who transact on
our omni services, which means including the website, including the
stores in a week. So that's the amount of foot traffic or
transactions that we see in a week. So if you think about scales
like that, any kind of disruption could cause
a massive amount of damage. So that is something that we wanted to sort of
think about when we even thought about doing this kind of journey
at Walmart. So to begin,
initially, we had to establish some truths, right? So we said,
okay, the following things are true. One,
reliability is no longer just a function of redundancy
and overscaled hardware, right? So we are not going to just throw some servers at
it and we'll obviously fix the problem.
That's no longer the case. Specifically the way
that we were thinking about it. We wanted to exist in a hybrid cloud environment.
If we wanted to do that, we have to acknowledge that external cloud providers,
no matter how much their guarantees could be, they are still a dependency,
a variable. And if something bad happens, we still have to be ready and
resolve essentially what the customers needs are.
Customers expect, do not
like the idea of scheduled downtimes, right. There was a time when that used
to happen. This is no longer the truth anymore, which means customers expect
you to be on all the time, have the best service,
doesn't matter how many parallel connections are open, how many parallel
transactions are running. They want everything to be just
as smooth as it's running at peak as it is running at non
break hours. Right. Any user that's actually performing
any transaction on the entire service does not
expect any loss of behavior.
Essentially like, your cart is not available, your items are not available, they want
it all the time. And that's the expectation. And that's not wrong.
That's how our customers are today.
And a direct correlation from that is the users could lose trust
on a brand because of a single moment
of a bad experience, a glitch, something bad happening in the back end.
Right? That loss could be temporary, which means they feel, oh, yeah, this is not
a good place for me right now, and I'll go back and come back later.
Or they could be like, lifetime, this is it. I don't trust this brand
at all. And they could communicate that to their families. You could lose their entire
business for their entire lifetime. Right. These are pretty big numbers that we had to
consider. So these are truths. We held them as truths. And using this, we said,
okay, fine. So what is the goal? So obviously you will see a
lot of connections to the principles of chaos engineering directly.
But this was our goal to maintain an application ecosystem
where if there are failures in infrastructure and dependencies, they cause
minimal disruption to the end user experience. And what is minimal disruption
is something that we sort of defined over time. We refined it from being
a very macro level, very sort of amorphous
detail to making it more and more fine grained over time.
Right? So the
first thing that we started talking about is, how do we inculcate the idea of
running a chaos exercise? Right? So we said, fine, let's talk
to them about an outage. You had an outage.
There was probably a lot of things going wrong. How did you manage
it? So we realized that by talking about it. We realized that
every outage was essentially a chaos exercise that was completely unintentional,
but was happening nonetheless and was causing people to be reactive.
It was exposing gaps in our systems. It was exposing things where we
were not as good as we could be. Obviously, the revenue impact
could be huge, but essentially this entire thing constituted
a chaos exercise. So you were sort of in that mode already.
The only idea was to say, let's not have it unintentionally.
Let's prepare for this from the start.
Right? Obviously, the measure of all
of these exercises that we did at the start was we said, okay, we want
to calculate what the downtime looks like. Right? So we wanted to calculate a
per incident cost. Any incident that happens, we want to calculate exactly how
much it costs the company. We want to break it down by
quarter and see trends, find exactly where we are
super efficient, where we are not. And then we wanted to track
this and find sort of a path, essentially,
right. We want to plan better a culture of software resilience. Instead of saying,
yeah, just fix this for now and it'll be fine.
So when we started doing calculations for an incident. So this should
be familiar with anyone who's dealt with incidents, there is a path, right?
You identify the incident, you page out,
there is an on call that receives it, they file an issue, or maybe your
l one supports file an issue. There is some
logging that also is sending out some alerts. There is
initial triage with the folks who are on call. They try out some stuff.
They figure out exactly what the playbooks look like. They will try some things out.
If that doesn't work, you will assign this to a subject matter expert
for a fix that could take some time. You could escalate if necessary.
Finally, you would resolve it, you would close it. If you find a bug,
you would deploy it, or you would do a hot deploy. Or if there is
something that requires you to rearchitect something because fundamentally something is wrong,
then you would write that feature in and then you would release it.
Right. This is expensive if you look at it.
We want to try and solve problems or reduce the costs in
such a way that we want to stop it at that third block.
We want to stop it right. Where there is initial triage, we want to have
enough experience for that on call person, enough information for that on call
person to be able to solve it at that third level. Right? So this is
what we called an incident cost, and this is what we calculated for all of
the incidents that we had. So once we had done this
as the initial step, we still had a long way to go because we had
to do a lot of homework. The teams that were running
this had to do some homework. Right? And this
is where I would start talking about how others could
also institute this in your companies. Right.
The homework that is needed is very generic. Everyone who's trying to
run chaos exercises should be doing this. First is
observability. This is a non negotiable. If you
do not know how your system can.
How. How can you sort of keep a pulse on the system and understand if
something goes wrong and identify it in a certain
specific kind of period, then you are not observable,
right? So obviously everyone relies on logging. There's a lot of ways
to do logging. We have splunk dashboards, cabana dashboards,
things like that. But that's not enough. That's only solving part of the
proposal. Right? You need to have intelligent second level metrics
to figure out exactly what challenges, right? You want those deltas,
you want those trends and the changes. You obviously
need alert setup. But I would say the last thing is
important, right? So if an on call engineer, the measure is if at
any team that you run, can an on call engineer who gets
on call for an incident resolve an issue successfully within
the SLA of a p one, right? Typically, the slas for p one resolutions
are short of acknowledgment, incident resolution.
If that person can essentially find the issue root,
cause it to a successful degree, and then
push in a fix within that time, that, to me,
indicates that there is significantly good observability in the
system. Right? Of course, it's not perfect.
I'm sure there is better ways to measure this, but this is something that I
think gives us a good measure of how the system
works. The other
on call prerequisites is imagine
if you're an on call, and if you have a team has an on call
and they wake up in the middle of the night. Let's say they wake up
at 02:00 a.m. And they're on a call with the VP and
the CTO, and they're like they're losing millions every second.
How can we fix this? Right? At that point, you do not want them
stumbling to try out things. You want them to have a specific playbook
for all of the issues that can go wrong, right? So you need to have
a disaster recovery playbook, and you need to have this playbook well tested.
Having a playbook is not the answer. Testing a playbook continuously is
the answer. Right. You need to make sure that for every application or every microservice,
the way you define it needs to define its own critical dependencies.
Like what are the dependencies which, when go down, impact the application
to a degree where it's not functional, it cannot do its job.
Those are critical dependencies, right. For every critical dependency failure,
there needs to be a detailed playbook or an automated way to
have fallbacks instituted whenever the critical dependency is
not available. Non critical dependencies also have to be
defined, right? It's possible that you are sending off to
a log that's like asynchronously sent to another service,
but it's possible that initially the failure may not
affect you, but over time, accumulation of those logs may
impact you in some way. Right. Maybe the space on your system
is being eaten up and eventually run out of disk space. Right.
That kind of non critical dependency needs to be defined along with the thresholds
of when that non critical dependency will start impacting a service.
Right. If you have this, then an oncology engineer, even in the middle
of the night on a call with a high pressure situation,
will be able to navigate this in a systematic way.
So that's what we think is required homework.
Because we realized that anyone who sort of solves goes through this
exercise, realizes a lot of gaps in the system automatically, even without running
a chaos exercise, right. This reveals a lot of gaps in the system.
So these are the two sort of key homeworks. I would say if you do
not have observability and you do not have these prerequisites, you will not be
able to get the most out of a chaos exercise.
The other thing I would say is there needs to be a way for you
to be able to understand your production load, what it looks like,
and be able to generate it in a sensible way. Right.
This is crucial for two reasons. One,
you do not ever want to run in prod
unless you really are confident. So you have to always have
a do no harm approach. Right. You cannot knowingly cause harm to
prod, so you want to know how to do this in a pre prod environment,
but using prod traffic so you can be confident of the results.
There is no point of running failure injection tests if you can't really verify
what will happen if there is a failure and what happens to prod traffic.
Right? And you want to try and do that as much as possible early enough
in the cycle, so that you're not causing the company expense.
By all means, if you're confident, you should be pushing your testing to prod as
well. But if you're not mature enough, this is something that's important,
right? The build or buy question.
This is something that I think has been answered by lots of folks
that have more knowledge about this than me. I don't think there is any preference.
If you have a system that can generate production like road internally,
that's great. If you have proprietary needs to do that, that's fine. But there is
also buy options which other teams and other companies have used.
Right? So I don't have a say in that, but that is something that has
been debated. I don't have a say in that much anyway,
so the other thing I would say needs investment is a CI
CD workflow, right? The diagram that I
have on the screen essentially demonstrates what a real CI CD pipeline looks
like in this day and age. There is a lot of automation.
You'll notice some of the stuff is very familiar to you, right? Which is plan,
code, build, test and deploy. Right. In between there
is a profile stage, right? A profile could be something like run a
performance test, find out exactly how much utilization I've been
using, recommend the best solution for me to exist in.
Like give me the right kind of cpu and memory allocations for my Kubernetes pods.
It could be give me the right kind of flavor for my vms on Azure,
whatever those are, right. That essentially has now come part
of the CI CD cycle. And having that enables you to
solve a lot of these reliability issues way earlier in
the pipeline, rather than wait for some signal to come out of production.
You will also notice there is two key things here, which is
there is a multi cloud environment. So hybrid cloud, I obviously represented that.
But there is two things. Inference layer, right? The inference layer is essentially
a back channel out from prod coming into the pipeline,
the CI CD pipeline. This is something where I say observability is important.
If you have an observable system, you can read what's happening in production
and then feed that back into to make your code do
better, right? The other thing you'll see is a decision engine that's between
the CI CD pipeline and the clouds. This is something
that many teams have started investing in, many big companies
have started investing in, which is figuring out what's the best
kind of cloud configuration,
if you will, for my application. Would it be better on
prem because it has latency restrictions? Would it
be better on a certain kind of cloud provider? Because certain kind of cloud provider
provides a certain kind of platform service? These decisions
have now started to get automated, right? And you'll see more of this coming in
the years. But this is an investment that I feel enables you to
have a better system in production.
Building a maturity model. So we did this at Walmart.
We did not allow teams to just go come in and run chaos engineering tests,
right? We have a detailed maturity model from level one of
maturity to level five, with requirements at each level. And this is all detailed
in that blog post. It talks about
how as a team, you can enable yourself to grow from one level to
another. And that also enables the team to become more and more confident.
It enables management to be more confident on the team running these exercises.
So that maturity model I have seen, it really helps. Right.
What happens essentially, because of that maturity model, is that over time,
as you go from red, which is level one, to green,
at level five, the support costs go down. Right.
What also happens is that the revenue lost per second, per minute, however you
calculate it also goes down, the resiliency obviously shoots up.
At level one, I would say the support costs could be in the five
digit number, five digit dollar number per minute, whereas at
level five, you're probably looking at a few hundred dollars per minute. Right. So imagine
a couple of engineers working on observability systems and just getting
all the answers at level five because they have everything in place, whereas a
whole bunch of team of engineers trying to figure out what do I do next
and what do I shut down to solve this? The other
thing is build the right tools. So we invested a lot of time in making
sure that we have the right tools to enable our teams to run the resiliency
test. One of the tools that we did invest in is resiliency,
doctor. You can read all about it in the
article that Vijita has published. So investing in
tools is important because I believe the ecosystem is still
sort of getting up to speed on that killer product.
So we all need to sort of contribute, chime in and do
our best. So this is essentially something that I would say
all companies would. It would be good value for money.
And this has really helped us enable that maturity model in all the teams.
So apart from the maturity model for the teams, the other thing that I would
focus on also is building the right mindset of each and every individual.
Right? So what we suggest all
companies to do, because we've seen this succeed, is a lot of trainings.
Right. The path is not easy. The journey is
not easy. You have to make sure that people understand what they're trying to
accomplish before they do it. So trainings, certifications, brown back sessions,
all of the usual stuff. Making sure that you're building resilience after an
outage. Right. And resilience doesn't just mean software resilience.
And I know there is other speakers who will talk about this,
which is about human resilience as well. Right. You want to make sure
that the person who was on call doesn't treat this as a traumatic experience.
They actually learn from it because the postmortem itself was
blameless. You're not really pointing fingers. You're trying to figure out what
the system looks like and how to make it better. And teaching that
is also something that doesn't come naturally to human beings.
Right. We tend to want to find blame in someone, and blameless
postmortems is something that I think it really takes a culture
change in companies to accept that.
And the last thing is carrots, not sticks, which teams, you treat them with
rewards, not punishments. Basically, the idea is
you provide some kind of an incentive for folks doing these kind
of resiliency exercises. It's a hard thing to do. The fact that they're
accepting it, doing it means that they're committed to it, they're passionate about
it. You want to make sure that you enable them and you incentivize them.
So what did we learn on our journey? So all of this stuff, I think,
is something that I think works for any company anywhere.
But there are some things that I would say do not do, and that's
what I'm going to talk about next. So these are the things that are don'ts.
So these are learnings. The first learning that we found was
we mistakenly created vanity
positions, right? And this is something that other folks also have done,
which is we did not want those, which is like, this person
is the designated chaos practitioner, or this person is the resiliency
expert, or such. And such that doesn't work
because, one, it shifts the responsibility
of something to be done to that one person instead of democratizing
it to everyone, saying, oh, yeah, we are all enabled and empowered
to be able to do this. It shouldn't be this one person. So we
learned that pretty early on, but it was something
that was, we realized that it was not helping our
cause. Second thing was, the exercises cannot
be conducted without complete participation. When I say complete participation,
it doesn't mean complete participation of just the team. It has to be the
team's dependencies. Maybe the SRE teams,
maybe the other teams that support you in production. All of these
folks need to have sort of, they need to be signed
up, they need to be committed to this cause. And that also takes effort.
It does also take interest from their side and passion from their side.
So this is something that we sort of found out over time, and we realized
that we have to sort of fix this. The other thing was
don't assume, verify. So this is sort of a
take on the same thing, right? So like trust but verify kind of thing.
But it always starts with assumptions, right? All of us in this room,
we know what the value prop of chaos engineering is, but you can't
go in with that assumption with everyone, right? So, for example, observability,
you cannot go into it if you ask a team, hey,
do you think you can find an issue in 15
minutes? They'll probably say yes. I don't think there is any team that has said
no to us or anyone else. Right. But you
have to verify that by checking if the observability truly is as
they say it is. Right. No. Teams, I would say,
is as well instrumented as they think they are.
On call. Being reactive is always the problem.
Right. You have to make sure that you test the on call systems
you want to test. Make sure that the on call person has everything. Doesn't mean
actually test the on call person to see like, hey, I just created an issue.
Did you actually catch it? Right. Not that way, but you want to make sure
that they are empowered before they get into a bad situation.
The other thing is specifying what the
team's goals are about. Resiliency is crucial.
Every team does not see it the same way.
It will change team to team, it will change application to application.
And those goals would be prioritized by that team. It is not something that you
can centrally prioritize. You can prioritize. Loss of revenue
should be minimized, but that is interpreted by different teams differently.
And I think this is something that we learned, and it was a pretty painful
learning for the simple reason that we didn't really understand
that teams had a certain plan as well for how resilient
they want to be. Instead, we were imposing a certain metric which for them was
meaningless. Right. So that's something I would suggest folks to have discussions
about. And obviously their deployment pipelines have to be verified.
But this, I would say, is something that is part of the exercise.
So are teams ready for exercises? So this is something that
no team is going to come in and say, if you just start your chaos
engineering group and you go to someone and you say, okay,
are you ready to do this? They are going to be very reluctant.
Maybe someone will say, yes, we will try it, but in reality,
none of them are right. So you have to make sure that you build the
training, you teach them that it's okay to sort of learn
but don't fail in such a way that you sort of cause that trauma
to the team saying, I'm never going to ever do this again, this was a
bad idea. So in order to teach the right way to do things,
you have to treat chaos engineering as it's supposed to be. It's an experimentation process.
It's a process where you establish a hypothesis and you
prove or disprove it. It's a scientific experiment. It's not a,
let's just hammer out some of these nails into this server
here, and let's see what happens. That's not what chaos engineering is. So I just
want to give you guys a sense of where we are now.
The report card reads really good. I would say that
all of these learnings that you saw, that has obviously given us a lot
of thought about how we want to progress in the future, but it has only
given us more sort of will to do more. The application
teams are eager to run these tests, and because of the maturity model,
they've understood that we do not have to just go in and be glamorous
overachievers overnight. It has to be a slow process.
Management confidence is good because management
invests in this.
Again, in order for management to invest in this,
they need to understand that chaos is not the goal. The goal
is resiliency, but the way to get to that is chaos. Right. And this
has been repeated by many folks, not just me. There's other speakers
also who will repeat this.
Increased resiliency in engineering usually tends
to basically, if your engineering team itself is resilient,
then essentially it opens the way to subsequent learning. So you start
doing more learnings. It encourages them to test things out even more.
And frankly speaking, everyone defines an end state. So application owners,
all of the application owners define, okay,
if something bad happens, this is what I want, right? That's all in their dreams.
There is no verification. But this now allows them to stand in meetings
where maybe there is multiple high level tech leads or senior management
and be able to say, okay, this is what my application does, and I'm confident
about it. And that's actually, I think, the best thing to come out of
it, because you want to empower your tech leads, your engineering managers and such,
so that they can stand their ground whenever they are talking to folks about
how to do this thing better or whatever. Right? Like, good design, good architecture,
all of that is improved by this. So this is where we are today.
And so all of the learnings that we have done has led us to this.
And going forward, what we want to do is build on these
models, right? The maturity models that we see today that I
just described, those were rough around
the edges, right? So there were things that we had not defined very well.
Like I said, resilience of the individual person as well.
Right. The engineer. So we are trying to work on those things and make sure
that we commit to this in a way that is sustainable
and that's actually important for this kind of exercise.
That's the entire story that I wanted to share with you guys today,
and that's all I had. Thank you so much. If you have questions, I'm ready
to hear it.