This transcript was autogenerated. To make changes, submit a PR.
We have a really interesting topic here. Most of these day, we have been
talking about chaos engineering in general. This talk is
a little bit more about how to practice it, rather than
preaching why is it needed and what it
Before we start, about me, I go by Uma
Mukara. You can call me Uma. I'm co founder and
CEO of a company called Mayadata.
We do cloud native data management primarily
for Kubernetes. So you can see
me talking a lot about Kubernetes today.
And I'm a co creator of the following open source
projects. The genesis of this entire
cloud native chaos engineering is really
about trying to do chaos engineering for
an open resources project called Open EBS.
And we ended up creating a
cloud native chaos engineering infrastructure called litmus.
While trying to do chaos engineering as a practice for
open EBS. I'll talk a little bit more about that.
So we've been talking chaos engineering.
It's a need everybody knows, right? So Russ Miles
started with why chaos engineering is a need.
And we've been also hearing it's a culture you need to
practice. It's not a thing. Right,
but I got all that. How do I actually get started?
Right? This is the question that we got into.
I'm in this Kubernetes world, and I'm
developing an application, or I'm operating a big infrastructure
or a deployments. I need chaos engineering. How do I
do? Right. As part of the operations work
I do, I got into
a situation where we need to practice chaos engineering for
a very different reason. We were building a data management
project called Open EBS, where the community
has to use this software
for relying on these data. Right.
Storage is a very difficult subject. We all know that.
So you don't want to lose your data at any point of time. So how
do you convince your users that you have done
enough testing or it has enough resilience?
So the best way is for them to find out that,
okay, I can actually break something and still see that it is good.
Right? So we started working on that and
then realized, okay, Kubernetes itself is
new. Chaos engineering is as an awareness
is growing, but we need to start creating some infrastructure
for our own use. And then slowly we realized this
can be useful for community as
a whole. And then we created litmus. Right?
So today I just want to touch upon the
following topics. We all know what is chaos
engineering, but what is cloud native chaos engineering, right? And then
if we understand why cloud native chaos engineering is
a different subject or a little bit of a deviation from
what we've been talking, what are those principles? Like?
We commonly observed some principles that we can say that here
are the cloud native chaos engineering principles.
And then we created a project called litmus, and then we can talk about
is it really following all these principles?
And then how many of you know operator
hub in Kubernetes
so very similar? Or helm charts, right. Helm charts
is a popular concept because you have a chart to
pull up and taken bring an application into life.
So same concept applies to a chaos experiment
as well. Right. Then we'll go to some examples and then we
can talk about what are
the next steps and then so forth. Right? So what is
cloud native chaos engineering? Are there any
cloud native developers here?
Yeah, what is the cloud native developer?
If you can Docker pull a developer,
then you're a cloud native developer,
right? So it's really about practicing
chaos engineering for specific environments,
which are called cloud native environments. We'll just see what
these environments are. And you can also say that
chaos engineering done in a cloud native way is called cloud
native chaos engineering. Even further, to simplify
chaos engineering done in a Kubernetes native way is
called cloud native chaos engineering. So it's all about this general concept
of how Kubernetes is changing our lives, has developers
and sres, and you might have observed this
many, many times that there is a particular way that
the DevOps have know the way we practice
DevOps have changed.
Adrian talked about infrastructure as a code
that has taken as a primary component
of Kubernetes deployments. Right.
So I'll just touch upon
what a cloud native environment looks like. I've taken this
concept or picture from one
of the GitLab commit conferences where Dan Cohn from CNCF
was presenting about why a CI pipeline is
useful in cloud native environment. I just
will repurpose that for a different need here.
Right, so you are writing a lot
of code if you're a cloud native developer,
and then after you develop, you run it through CI pipelines and then you put
it onto production, or like you release it and somebody's going to pull
that and then use it, right? So where are they going to deploy it?
So they're going to. Let's assume that it's a Linux environment, it's a
lot of code and you will have kubernetes
running on it, right? That's much more code than Linux
itself, right? 35 million of source locs
compared to the Linux. And then you will have a lot of microservices
infrastructure applications that will help you run your code
and then there are some lot of small libraries or databases that
run on top. And finally your code, right?
So that's how small your code is, it's about 1%.
Right? And the common factor is you need to think how often
your Linux upgrades happen,
right? Maybe 18 months, twelve months.
And can you guess how often Kubernetes upgrades happen
before you know, you understood one version of kubernetes, another version comes
in, right? So it's very dynamic and
that's the purpose of actually adopting kubernetes because it runs anywhere and
then everybody runs one thing inside
a container in one thing only, right?
So kubernetes upgrades happen very frequently.
And your apps, not your apps,
but the application or microservices surrounding your
application, these also need to be updated very
frequently and then you don't control all of it, right? And you are
always thinking about my code. How do I make sure that the
code that I bring in deployments is working fine,
right? So the summary is that your environment, the cloud native environment,
is very dynamic and then it requires a continuous
verification, right? I mean it's the same concept of
don't assume but verify. Right? But what do you
verify? You verify your 50,000 lines
of code or remaining 99%. How do
you verify? So you cannot go and say that
everything was working fine. I verified everything if I'm an
SRE. But it's a bug in the Kubernetes
service that is provided by the cloud provider, right.
You cannot assume that cloud provider has
really hardened up the Kubernetes
stack or any other microservices. So it's really up to the
SRE or running whoever is running the ops to verify
that things are going to be fine. Right?
So that's one thing. So how do you do this?
Obviously it's chaos engineering. Chaos engineering,
not just for your code, but for the entire
stack, right? Because your application depends
on the entire pyramid. So that's
one concept. The other big difference is, okay, I need to
do chaos engineering, but how, what's this cloud native
environment? The big difference is everything
is yaml. You need to practice this yaml
engineering, right? You cannot write. I mean,
I think Andrew has really demonstrated very well today about
how to do chaos engineering for MySQl server
on kubernetes, how we did like
we really killed some pods or cube invaders. That's good for demos.
But how do you really practice, if you are necessary in production,
right? You have certain principles. If I am putting something
into an infrastructure, it should be in a yaml,
I need to put it into git. And then even if I'm
doing some chaos experiment, it should be in a git because
I am making an infrastructure change, I'm killing a pod.
Who gave you permission? Right? So it has to be
recorded and that's Gitops. Right?
So that's what is the cloud native environment
dictates? If you are a cloud native engineer or a developer, you have
to be following this infrastructure principles.
So what you need is not just chaos engineering,
but it is cloud native chaos engineering. That's the concept I
want to drive here in
a simple way. This is my definition.
If you try to put chaos engineering as
an intent into a manifest AMl manifest, then you can
say that you are practicing cloud native
chaos engineering. So that's how we started, right? So we are going
to give open EBS as an application to your
users and they're going to take it, run it. How do they verify?
However they've been verifying
an application is same as deploying an application. How do
you deploy an application? You pull up some resources
into a ML file and then you do a Kubectl apply,
your application comes. How do you kill it? It has
to be the same way, right? So that's the primary difference that
we found. And then we started building up this infrastructure
pieces to do the same thing, right?
So let's look at Kubernetes Resources that we have
We have a lot of resources provided by Kubernetes as an
infrastructure component, right? Pod deployment, pvcs services,
stateful set. And you can write a lot of crds, right?
So you use all those things and then define an application,
you manage an application and then this concept of crds
have brought in these operators. You can write your own operators to
manage the lifecycle of the CRS
itself, right. That's good for development. If I'm
practicing chaos engineering or chaos testing for
kubernetes, I need to have a similar resources.
So I need to be able to have some chaos resources.
So what are those chaos resources that I would need?
So you would need some chaos CRS in general,
right? So just like an application is
defined by a pod and a service and some other things. And how
do I define chaos engineering? You need to be able to visualize your
entire practice through some CRS. And obviously
you need a chaos operator to manage the CRS and to put things
into perspective, upgrade each test,
right? And it's all about observability.
If you don't have observability, you can introduce chaos,
and then you're not able to interpret what's going on.
And then it's all about history has. Well, right. So metrics
are very keen in chaos engineering.
So these are the chaos resources that we figured out at a general.
These three resources are needed. Right? So then
looking at all those things in general, we can summarize. The principles
of cloud native chaos engineering are the following,
right? So first of all, whatever you try to
do, it has to be open source, because now we are trying to generalize.
Right. I can just write some closed source code and say that these.
It is. You're not using to accept it. Right.
Kubernetes was adopted and then the entire new
stuff is becoming adoptable so fast because it's open
source. So that's just a
summary of it. And then you need to have acceptable
APIs or CRS, just like pod service for
chaos engineering. And then what's
the actual logic? How do you kill something, right? So that,
again, you should not determine. It should be pluggable. Right.
Maybe my project does it in a particular way,
but somebody else also has a particular way of killing. For example,
here today, we talked about Pumbaa, right? How do
you introduce network delays? It does in certain way.
And then you should be able
to use Pumbaa and plug it into this cloud native
chaos engineering. And it should be usable and
then it should be community driven. Right? So when will you
do this chaos engineering as a practice? It's not just preaching
about chaos engineering, that it is a culture. It's a good thing
to do. What are the best practices? We all need to be able to build
these chaos or experiments together, right? And that's
when you call it as principles. Right? So I've written a
blog about whatever I just said and then published it on CNCF,
same concepts. And then it's available on
CNCF itself. Now what I want
to do is with these principles, this is
how actually we grew, started practicing chaos
engineering, and then we named it as litmus. And Litmus
project is exactly a manifestation of our effort to
practice chaos engineering with kubernetes. And we turned it into
a project that it's not just useful for, only for our project,
but anybody who is practicing or
developing on kubernetes want to practice chaos engineering,
they can use litmus. Right? So this is just a brief
introduction. It's totally apache two license.
Then there are some good contributions coming in.
It recently went GA 10 a couple of weeks ago.
It really means that it's got all the tools
or infrastructure pieces that is needed for you to
start taking a look at it. And then it's
open source, obviously, it's got some Aps or CRS.
I will explain that in a bit. And then whatever
the chaos logic that you're using, you can just wrap it up into a docker
container and put it into litmus, right. You don't need to change anything.
And then it is obviously community driven. I'll explain
that in a bit. So let's see, what are the CRS or CRDs
that litmus has? Right? So these first thing is chaos
experiment. You want to do something. Take the
minimal thing that you want to introduce as a chaos and define that
as a chaos experiment, right?
You can do a killing of a particular application that might
include three, four different chaos experiments. But a
chaos experiment is something that is like minimalistic
killing of something, right? And then
you need to be able to drive this chaos experiment.
For an application, we call that as chaos engine. This is
where you tell. Here are the chaos experiment. This belongs to this application.
Here is your service account. And then who can kill, et cetera.
So you define that. So to do this, and then after you
do, you need to be able to put your results into another
Cr called chaos resulted, so that Prometheus
or some other metrics can come and pull it up, and then somebody
else is going to make sense or some tool is going to make sense of
what exactly has happened, right? And then there will be multiple
chaos experiments that you can keep adding into a chaos engine,
right? So that's the CR that
you have. And then for pluggable chaos,
for example, it already has Pumbaa and powerful seal as
built in libraries. You can actually pull in your own library.
So how do you pull your library into this infrastructure?
For example, here I'm explaining powerful seal.
How we did it, the community did it, is all you need to do
is whatever the killing that you do, you put that into a
docker image, right? So if you just do a docker
run of that image, and if it goes and kills something or introduces
chaos, then it can be used with this infrastructure,
right? So once you have the Docker image, you just create
a new CR, new experiment.
A new experiment really points to a
new custom resources. And then inside that custom resource definition,
you just say that here is your chaos library, right? So it's
as simple as that, right? So the
reason why I'm just trying to emphasize this is think
litmus as a way to use your
chaos experiments with a more acceptable
Gitops way you can just orchestrate
all your experiments using litmus. The way they are written,
you just need to convert them into a Docker image, right?
So litmus chaos render automatically picks up this docker image and
then runs, puts it into a chaos result cr and then you're
good to go. And then these community is developing more tools
to observe chaos. All that will work very naturally.
Right? So it's community driven. What it really means
that we have something called operator hub, a chaos hub.
It's got a lot of experiments already and taken you
as a developer when you create a chaos experiment.
If you want this chaos experiment to be used in production or
in pre production by your users, you're going to push it into your chaos
hub. And your sres, whoever is practicing your chaos
engineering, are going to pull this experiment and then use it,
right? So imagine if Andrew had published that experiment.
You just need to create a yaml file and
then put in some key value pairs and then it runs.
So this is how the cloud native orchestrator architecture
looks like. So you will have a lot of crs
defined for your users, chaos users.
And you have a lot of experiments that are being used by
this CRS, but you can develop more, right?
So has more and more people develop this chaos
for various applications. You will slowly see that the entire
Kubernetes chaos engineering can be practiced just
by installing litmus, right? So how
do you get started? Just imagine that you have on the hub a
lot of chaos experiment for various different applications
and you are running your app in a container and
then all you need to do is pull litmus
helm shot or just install it. It runs in a container.
The moment you install you will get chaos libraries and operator onto
your Kubernetes cluster. And then you need to pull.
The assumption here is that you have so many charts and you
may not need all of them. So you pull whatever you need onto
your Kubernetes cluster and then you just inject chaos
by creating a cr, right, chaos engine which points
to various different experiments and then
it opens up, the operator goes and runs
chaos against a given application and it creates a chaos result,
right? It's as simple has that.
So let's see an example of how this changes
a cloud native developer everyday's life,
right? How does a developer
create a resource? Right? You create a pod and then you
create more resources for an application, for example a
pv or a service, et cetera. And that's
usually that's where it ends. You want to test something,
right? So how do you do it, you inject chaos
by simply creating one more crs just like you've
been using kubernetes for, right?
So you can actually create a chaos engine and
tell what needs to be killed where and then it's all done,
you get your results. So it's extending the concept
of your experience with kubernetes to
do the Chaos engineering also. So that's cloud native chaos engineering.
Just to summarize. Right, so on the Chaos hub we
generally have two types of experiments.
One is generic, that is like generic chaos experiments for
Kubernetes resources in general. And then there are application
specific. So this is where it gets interesting.
So you can, like we have seen again,
sorry to take these same example again and again
Andrew showed I do container
kill of MySQl server pod.
Right. So then you have to go and verify
what exactly happened, right. We verified that the pods have come back.
That's all we saw. Right. In Kubernetes invader.
Hey, more pods are coming up. But how do we verify whether
application that the MySQL server is
really working well or not automatically. So that's these,
you can write more logic into your application specific chaos experiment
and then use them in production. So some
of the experiments that are available are already available.
You can just do a pod delete, container kill.
I mean like a pod can have multiple containers, you can just kill only one
of that. And then you can do a cpu hog into
a pod. Network latency, network loss, network packet corruption
just introduce some corruption into your packets that
are going into your pod and see what happens. Right? So that could be
the granularity level that you want. Then there are some infrastructure
chaos. Of course these are specific to the cloud providers.
For example, how do you take down a node on AWS
is different than how do you take down a node on Google.
Right? So there are disk losses, which is
a very common thing, right? So I suddenly
lose a disk, what happens? And then disk fill is
one more common thing. So these are the initial set of things that are already
there, you can start practicing and application specific things
that are available here are an
application really constitute a pod
or multiple pods and then it's got some service and then there's got some
data. Right? So what do you mean attacking
an application? Right? So you need to define the logic
of what is it that you're going to kill.
Am I going to kill MySQl server or am
I going to kill a part of it, et cetera, et cetera. That's definition.
And then you verify before killing everything is good or not,
right? So that's the hypothesis. And then you use the generic
experiments to actually do the chaos and then
verify your post checks, right? So all this can be put
in into an experiment, and every time, you don't need to run
it again and again. So just put that into Yaml file and your application
chaos happens, right? So the result Cr gets
done. So for example, I'll quickly take an example of open EBS,
how it's done. Open EBS has got multiple components. Now I'm verifying
open EBS as an application. Works well when I
kill something, right? I cannot go and say, okay, open EBS
is a cloud native app. That means it's a microservices and it's got
multiple pods. I can kill a
container that belongs to openabs and see what happens. That's one
way of saying. The other way of saying is I can kill a controller
target of openabs and see what happens,
right? So you will end up having multiple different
chaos experiments that are specific to that application, and then
you can go start using them. For example, I can kill an SQC target of
open EBS and then see what happens. You don't need to really know what
should happen. It's all defined by the open EBS developer. As an open
EBS user, you will be able to say,
okay, my open EBS is functioning properly because I just killed
a target and then it is behaving as expected.
Or you can kill a replica and then see, so you don't
need to learn the nittygritties or complexities of the application,
how the application should behave when something happens in production.
So all that is coded up by your developers and pushed up onto
chaos hub, and then you can just use it. Right? So that's
summary of how a cloud native chaos engineering
framework can work, and litmus is just one of that, then you can
contribute. It's on the Kubernetes slack itself. And then
if you find some issues, you do that. But primarily take a look at,
if you are practicing chaos engineering, take a look at chaos
hub and then there are some things that are already there and it's
just these beginning we opened up. And then
hopefully more contributions will come from the community, from the