This transcript was autogenerated. To make changes, submit a PR.
Sometimes being the last talk. Actually being the last talk is always super
dope. But there's different reasons, right?
These best thing about being the last talk is you can kind of tie things
together. You know, I can listen to all the other talks that happen
throughout the day and then tell the story.
The other thing is sometimes all the things you wanted to say have already been
said. So talks over. We're done. Be. Let's go. No,
no, seriously. I'm using to try to be a little flexible
as how we're going based
on these ideas of things that we've already talked about.
So going to take this opportunity. I've got some.
That's weird. Well, sorry. On that screen,
just noticed that you can't really see the top. Well, all the really good
information is at the very top of the slide, so you're missing
all the real thought leadership over there.
This talk is called the psychology of chaos engineering.
So it's thinking along the lines of a lot
of the human factors that come into play. My name is
Maddie Stratton. I'm a DevOps advocate for
Pagerduty. So I'm not a huge fan of resume slides,
but I really like this one. So you're just using to have to sit through
it. It's cool. So anyway, I work for Pagerduty. How many
people here have heard of Pagerduty? You already heard about it in some earlier slides
today, too. So if you didn't raise your hand, you're lying. Cool.
That's as much as we're going to talk about Pagerduty right now. Business. It's the
bottom of every slide. I do a podcast called Arrested DevOps.
We're one of the longest running, still running DevOps podcasts.
If you're a listener and you want a sticker, come see me later. I got
lots of stickers. I also have cool pager duty stickers, too, because Deverel
life is giving away stickers. That's my real job, sticker engineer.
I founded DevOps days Chicago, which is going
to be in September. So if you're into DevOps and you're going to be in
Chicago, you should come to it and we'll talk about it later. And I help
run DevOps days all over the world. And this
used to be my license plate. So one might say I'm
invested in DevOps. But then DevOps is like ten years old. It literally
is, right? Like, the first DevOps days was ten years ago. I want
to be cutting edge and everything, so I actually had to get a new license
plate. So this is my current license plate. That's true,
by the way, that is my car.
So I might have $200 more than cents
might be because that's what a vanity license plate costs in Illinois.
But in reality, what I like to do at this part of
the talk is kind of get a level set and get
some agreement so that we're kind of speaking a
similar language. And in the case of this being towards the end of the day,
we've seen all these talks. Fortunately,
the level setting that I wanted to do has already been done.
Right. Nobody really said anything today that concretely disagrees
with anything I was going to say, so thank God, because that would have been
awkward. But a couple of things that I do like to, like to
kind of stress and you may find. So the nice thing, but doing
this talk at the end of the day, especially at these conference, is that the
things that you already know, those are review. These is all.
It's like it's on purpose, right? We're tying it all back together.
So, first of all, also, I don't always give this talk
at chaos engineering conferences. So sometimes I have to tell people what I
think chaos engineering is. And by I, I mean somebody else's definition that I like.
So this is from principles of chaos. But really, the couple of the things
about that that I always like to kind of bring. But we talked about experimenting.
I think we're all pretty common and thinking about that, and we're looking about
building confidence to be able to withstand these turbulent conditions.
Right. You'll notice there was nothing in here about prediction.
So that's just kind of my little thought. This isn't
an old definition, but we can go back here. I was just talking about.
It's almost ten years ago. So this was from these Netflix tech blog
talking about chaos monkey. But there's three things I like
to pick out of this definition that are
generally interesting. And again, think about at some conferences I'm giving
this talk, this is a brand new idea here. We're reviewing, we're getting on the
same page. So it's. Right. So they're running
elements in the middle of the business day. And this is
similar to at page of duty. We run our failure Fridays during
the day. They're run it at lunchtime, Pacific time, because frankly,
there's no good time for pager duty to be down. If there is any
good time for us to take any kind of an incident, it's during the day
in San Francisco when most of the people are in that particular
office. People in our Toronto office will
tell me that, no, the best time is during the day in Toronto because Canadians
are just as important as valley people. And that's totally true.
Carefully monitored environment is a big key part of this. Right?
And then again, having your engineers standing by. So these is all
stuff that hopefully in this room is like things we consider as table stakes.
If we don't consider this table stakes. First of all, if any of this comes
as a big surprise to you right now, you've probably been sleeping all day,
right? Which I totally get. So cool. But perceptions.
So here's the thing. We just talked about what we understand this to be,
but there's different perceptions around what
chaos engineering is and what it provides and also what's
involved. And sometimes people say perception is reality.
What you're trying to do is affect change in an organization, likely. So perception
super matters. It's not about being right. That's what Twitter
is for. When you're doing change inside your organization, it's about understanding
the perceptions. So this
has been alluded to before, but this drives me up a freaking wall.
Anytime I try to go somewhere and collect information about chaos
engineering, several people think they are the most clever person
in the world. That will be the first ones who ever make these joke,
right? They're not also to see the people who come up to the pager duty
booth and go, ha, I hate you. You wake me up in the middle of
the night, we're like, clever. Never heard that one before. So this is generally my
response to that. It's like, first of all, you're wrong,
right? That's why it's on social media. But also, it's not even clever.
So that's the other thing. It offends me. As Jerry Seinfeld
said before, to sort of paraphrase. It offends me. Has a chaos engineering
advocate, and it offends me as a comedian, because it's
not even funny. If you're using to troll, at least be funny.
Okay? And again,
it's not about breaking things, right? Our intent is
not to actually break something and try to
push it to its limit and go kind of, haha. I figured out how
to. But your system, right? That's a different thing. And if you don't believe me,
believe Sylvia. Because if you know anything about Sylvia Botros from
Sengrid, she is the expert at breaking things. And if Sylvia tells you
it's not about breaking things, these it super isn't. So that's got to be true.
Look, I know you know this, right? So why am I bringing
it up? One is, it's kind of a nice way to round out these day.
The other is we have to kind
of continually remind ourselves of these points and
these principles, because almost by virtue of
you sitting in this room, you're at a certain level of understanding
of this practices and of this field and the first principles surrounding
it. The people you're working with in your organization
may not be. And it's very easy, as we become
further advanced in our understanding of something, to kind
of forget where we came from,
not even necessarily where we came from, but where people at a different mode
of understanding might be. Right. So, again, I know you
know this I'm using to say it. Anyway, the good thing is
this just sort of blends right into the end of the day. So this is
almost just like bullshitting at the bar. We're kind of at a bar, so it's
cool, right? All right. So like I said, you know this, but they're experiments.
I think that's a really key thing, and it's a really key way
to help with that understanding. We've talked about hypotheses,
right? And our hypothesis should
be that if we do this thing,
if this condition exists, my hypothesis
is it will still work.
If. My hypothesis is, if I shut down this node,
everything's going to go to shit. I probably shouldn't run
that experiment, right?
Again, we're testing out assumptions and
hypothesis now. If your hypothesis is everything will go terrible, then, yeah,
maybe you still want to run it, but you definitely should run that in a
lower environment. That's a whole different talk to talk about the myth of staging.
So we're just not going to talk about that right now. That's cool, right?
So, again, taking a scientific approach,
I absolutely loved that convergence divergence
thing that Adrian talked about, and I'm going to steal it for the next iteration
of this talk that I give when Adrian hasn't given that talk right before it.
So, these. I will look like the clever one. I may
actually attribute it to Adrian. We'll see how that goes. So you're
like, we know this, Maddie. We got it right. And why does it
matter? Because how we talk about things matter.
And this is a little tricky. Sometimes words
matter, and sometimes these don't, right? So getting nerd
snipe points on Twitter for someone using a word wrong just
to be right is not helpful. So sometimes a word matters,
and a word matters when it affects how we think about something.
So I'm going to take a couple of examples, not directly
related, but to illustrate that point so we can see how it applies.
So before working at pagerduty, I worked
at chef, come from an infrastructure, has code background, so automate all
things. That's amazing. But the
components that make up chef code, one of
the elements of those are called recipes. Because of course they are. Because chef,
by the way, if you hate food puns, you definitely should go use puppet or
ansible, don't come to chef because chef is a t shirt company that
also sells software. And then we also make a lot of would
oftentimes, instead of calling it a recipe,
people, customers I was working with, users that are trying to adopt
this would talk about chef scripts. And that's
one that I would correct gently and with an explanation. Not because
I'm like, oh no, it's a recipe because chef and food and blah, blah,
blah. But it's a different way of thinking about what that actual application
of a concept is doing, right? A script is iterative
going through step. It's a stepwise thing. A recipe,
maybe it's not a perfect analogy, but the point is that recipe is
good and script is bad, but script is not the was we want to think,
so we're going to use a different word.
There's other things that I choose not to get a pedantic about.
I work at pager duty. We talk about post mortems a lot because incidents,
right? So how many people are aware that there's some people that
don't like calling them post mortems and for good reason.
Right? But it's not fundamentally,
by calling it a post incident review or a retrospective or an after
action report or whatever is not inherently changing how you think about
it. You can have a very good reason for not wanting to call it a
post mortem, but it's not related to a
change in behavior of how you do it.
How many of you follow me on Twitter? It's okay if the answer is no.
But if you do, you can probably take a guess as to what the next
word is that I'm going to say I'm picky about. And that's the word root
cause, right? These reason, first of all, I'm not getting
nerd points here. If you want to call it root cause, that's great. You're a
fine human being and I love you. But here's why I think,
contributing factor, why this one matters, because it changes how we think
when we use that word. It makes us think about a singular cause,
which in complex systems is not there. So my whole point of this,
this is not about stop using the word root cause, by the way, I stopped
using these word root cause. It's about the words we're going
to use when we're trying to affect change using chaos engineering within
organizations. Right. The thing is, people get
nervous. It's kind of how we live. It's kind
of how we've survived as a species is because we're worried about
risk. If we weren't worried about risk, we have all been eaten by antelopes.
Well, maybe not antelopes. I don't know. I'm not good at animals, but something bigger
with teeth. So risk aversion is kind
of baked into us. So we're going to get nervous about things that seem
to imply additional risk.
Right. You're going to do what in production?
How many people think about folks inside
your organization whose entire focus
and lens by which they view
their relationship to your company's business
is mitigating risk? There's a lot of people that. That's their
job, and that's great if
you are going to come to them and say, I would like to do this
thing, and they hear it as I would like to create more risk.
The conversation is now over. The irony
is that folks who want to, we know we're all sitting here going, but that's
ridiculous. Chaos engineering helps lower risk.
It's good. It's blah, blah, great. And they will love that once they understand
it. But you want to be able to have that conversation.
So Adrian talked about that a little bit before. Right? And again, this is beautiful.
Everyone's already given my talk, but this is why it matters.
And when we're thinking about mitigating
risk, I also like to mean, again, use your monitoring like it's for real.
We've had already conversations around this, your chaos
experiment, to be successful. And by successful, not to prove
your point, but it's not actually get you fired. You need
to be looking at the impact to production. You need to be monitoring like it's
for real. And it's because it is another way to
think about this. And we've seen some examples, but at pager duty,
we run our failure Fridays like a regular incident. We already
start incident response as part of the failure experiment,
and there's two reasons for that. One is we're already tuned up for it.
For us, it's also, we believe very strongly in always practicing incident
response. And a failure experiment is a really great
way to normalize the practice of incident response.
This is a little different. When Ross was talking about, like, a fire drill,
it makes you complacent. The difference of this is just being
users to the motions. Right. It doesn't mean because we do want
to try to reduce some of the stress. So, for example, the way we train
incident commanders, if you want to be an incident commander at pager duty,
before you go on call as an incident commander, you have to have run a
failure Friday because that's giving an opportunity to
practice that under a relatively low stress situation.
The reason I'm bringing this up is so we might take
this and say, like, oh, well, then a failure day. A failure
game day is a great way to practice a stressful situation.
It's a great way to practice stress. It's a great way to practice trouble.
No, it's not. No, don't do that. Because the first thing is your people
don't need training and practice of being under stress.
They get plenty of that already. Right. What we want to do is the opposite.
So, because we have some insight into what's likely
to happen, or at least the systems that are affected
and we know what we actually acted upon,
it's a really great way to practice and go through the motions, which create kind
of a physiological. So, you know, as Paige
says, something's broken, it's your fault. In this case, it actually is.
You can be a little blameful in chaos days,
but it's good blame, right? That's cool. Okay, so what
about the people? So we've talked about a lot about tech. We've talked a lot
about the systems, the technology, the providers,
the terraforming, the Kubernetes. That's all the fun stuff,
right? It's also the easy stuff. The humans pieces come in.
So when we think about the people that are involved, and there's lots of
people that come in, there's your employees,
right? There's your delivery teams that are for the systems that we're
running elements on. There's the people that are engaging in the experiment.
And, I don't know, you might have some customers or
users that might be wanting to use these systems. Those are people. They're involved.
Right. So I always kind of, like, ask this question.
If I said, how does it make you feel to know that someone like Netflix
is practices these principles? We're actually pretty down with it,
right? We're like, that's cool, man. They're Netflix. They're DevOps and all the. You know,
and that's cool. So I know that I can get
my new movies, and I can binge all the things
and whatever. Yeah. What about your bank again?
And everybody in this room buys into this, and this one
still made you. Hmm. Right.
But we totally know why it matters. And the thing is,
the blast radius is what matters.
I was pleased to see that Adrian also goes to Twitter like I do.
So I had a very scientific survey and said, if you discover
a service you consume uses chaos engineering production, do you feel reassured
or uneasy? And most people said they were reassured. This is not scientific at
all, by the way. And also there's a little selection bias,
but graphs mean results. So I had to
add some data and a little more data, such as it
is. I did a few surveys around words that people might
use to describe. So I thought it was interesting to say what words describe your
personal feeling towards use of chaos engineering on your team.
And a lot of people were optimistic, but there was quite a fair amount of
uneasy and cynical and everything. And then this is
what tells you everything. But engineers, when we flipped it
and we actually said, what about if products you use?
They were like, oh, my God, that's fantastic. But not us, because we're terrible.
Right. So I thought that was really interesting. But one of
the things that happens is when there's an understanding of
the effects, this can actually have a really positive effect on your delivery
teams, because we feel more comfortable making changes,
we feel we have greater trust in the system, but we
have to understand it because it has the opposite effect if we don't understand it,
right? So this principle that when someone thinks
it's about breaking things on purpose and all those things, it's actually going to reduce
their. It's going to make them feel uneasy, but when there's a greater understanding.
So this really boils down to education being helpful. And we
talked about people getting nervous. Management can get very nervous.
And we think about considering our words,
and this is usually the part when I say,
I don't have a great suggestion of what you should
call it business failure injection or chaos
engineering, because it might be different. Fortunately, a bunch of
people today have already given you a bunch of really good of.
And the thing that's important is accuracy is not the most important
part. So, like, Ross talked about system verification,
and the nerd snipes and us went, well, technically, it's not system verification
because that's a formal process. It doesn't matter. You're talking to your
CFO, right? Because all you want to actually do is get in the
door to talk about it, right?
Have that conversation. I like what Adrian says, just call it engineering.
But that doesn't work as well for when you're trying to explain a new practice.
So I invite everybody to kind of think on this, and I'd love to talk
about it if people have examples of ideas,
but it will vary depending upon your organizations.
Right. And really it comes down to the understanding of
the philosophy. And when you're trying to bring people along for a ride,
you want to be somewhat like minded. Right.
So I really like these. So Cody doesn't really use chaos
engineers fully, but had some interesting failures from
interns, which I guess is some form of chaos engineers.
And he says now, honestly, feeling excitement when
confronted with a new error that hasn't occurred before. This is really a
lot more about learning from incidents. But if you have those things that have to
be coupled and learning from incidents,
it's a thing that we're all not very good at. By all, I don't mean
all of us are bad at it, but I mean few of us are good
at it because we usually look at
incidents as something to be avoided. Again, we don't encourage
them. We don't want to say like, boy, I sure wish that I had a
ton and ton of incidents. Although one
potentially controversial way to say it is it's been said,
hey, you want to get better at incident response. If you tried having more incidents,
and that immediately sounds funny, but actually the way it's phrased is scope
more things to be considered as an incident, you will practice it more.
And so incidents are a gift. If that's a little hard to wrap your to
kind of swallow, maybe you say incidents are unplanned investment,
but if we don't think about, if we're not focusing on being a learning culture,
we're not going to get a lot of value out of all the practices we've
talked about today, it's about the learning. And that's why I thought it was so
great that we had to talk about post mortems as they apply to
that. We need to run all the things we would do for an incident on
our elements as well,
especially. But the only reason that makes sense is
if your after incident review, your pir, your post mortem is focused
on learning. If the whole reason you do a post mortem is
to write down these root case, it doesn't make any
sense to you to do it after a chaos experiment because you already know what
the root cause was, it was turned off, this thing. So those two
things, I think these practices are so well coupled which is why
in a lot of people's mind, they all kind of run under resiliency engineering
practices, right? Learning from incidents. These things are all loosely coupled,
right? They don't give you those things, but they all end up being connected
because they all come back to us wanting to learn and have
better understanding and be able to reason about our systems more
I'm going a little quick. Part of it is because we're getting to the end
of the day, and I know we started running behind, and I'd love to have
a little more hallway track and everything like that. But just a couple of things
I'd like to consider. Again, safety first, right?
And this is in the safety two idea of safety, but safety first.
We want to think about all sorts of things we've talked about today, about minimizing
blast radius, about making sure your responders, by the way,
just speaking of one thing I forgot, was I the only one who was waiting
for the last slide for Muri's? Talk about where we could, like where the git
repo was for Gabetta, and they're like, oh, no, we don't have that yet because
I want it. Right. So that was exciting to me because of,
again, the thoughts around the whole experiment,
right? So a couple things I think that are very key
to keep in mind, and these may seem like table stakes, but to
use a terrible analogy, I used to do swing dancing, and we
used to used to say about dancers, we'd say beginning dancers
take intermediate classes, intermediate dancers take advanced classes,
advanced dancers take beginner classes. So it's always helpful.
As much as we're big experts in this room, a couple of these things.
If it is common sense, great, it's a good review. If it's table stakes,
I assume you're already doing all of these. But knowing your conditions of when do
you shut down the experiment? And that's knowing what your key business
indicators are, right? You're not shutting down the experiment because
SQL server is using too much memory. All of a sudden, you're shutting
down the experiment because your average card size has dropped below where it's
supposed to be. Know what your key business metrics are that are these,
you're just going to call it. So you need to know what those are.
Right? And again, we want to build
resiliency, and resiliency comes from people having adaptive capacity,
right? And what we're not trying to do here is stress test
people, right? Even if Adrian's going to break necks of your
key sres, you're not trying to add stress,
you're trying to find challenges. So it
can be very tempting to look at a case experiment as a way to
test people's ability to troubleshoot or
to simulate the stress of on call. We want
to have transparency on everything that's happened. Never surprise anybody with your
chaos experiment. Right. And at these end of all these wonderful numbers
about reducing MTTR and availability, numbers that are happening,
everything, there are always people. And where I bring that up to mind is that
some of the scale at which we work, a small
number is not a small number. We're like, oh, well, we only
impacted one 10th of a percent of our users. Okay, well, that might have been
like 10,000 people that just had a shitty hour because
we weren't really watching those key metrics. So at
the other side of all of your grafana and all of this, there are humans.
And we always want to keep the humans involved. And thinking about these,
it doesn't mean we don't take risks. But remember that at the end
of the day, it's really easy to look at a graph, but all those little
things at the end are probably a person most likely.
So trying to remember the humans is what matters.
So, not that these slides are terribly exciting,
but if you want to check them out, that's my speaking
website. The slides are there, some supporting links, some other articles that
I found interesting. There's a couple of articles that I linked to in there that
I didn't talk about, that are about how we do failure Fridays
and stuff at page of duty, which is just not saying you should do exactly
what we do, but you're all interested in this domain, so it's more things to
know, right? And yeah,
if you like Twitter, that's where you can find me. I'm Matt Stratton, and Peggy
says, follow me.