This transcript was autogenerated. To make changes, submit a PR.
Hello, I'm Rafael and my talk is titled Mutation Testing
with Pete. But first, a few words about myself. I am a cloud native
team lead at Hazelcast before I worked at Google and
CERN. I'm also an author of the book continuous delivery with Docker and
Jenkins, and from time to time I do conference speaking and trainings,
but on a daily basis. I'm a developer and I live in Krakow in
Poland. A few words about Hazelcast Hazelcast is a distributed company
and it is distributed in two meanings. First meaning is
we are distributed company because we produce distributed software. But the second
meaning is that we are distributed company because we all
work remotely. We always work remotely. So I'm from Krakow,
from Poland, but I'm the only one from Krakow. We have people all over the
world. Our products are hazelcast in memory data grid,
which is a library for distributed computation,
for caching. Hazelcast jet is a library for stream processing, and hazelcast
cloud is hazelcast, but as a service. So do you
know why the NASA spacecraft burned in the atmosphere of
Mars in 1999? And do you know that it is somehow
related to the fact that even though mutation testing
was discovered in 1971, it was not really
used until 2012? So the answer to
all of these questions, as well as the whole idea and the implementation
of mutation testing, you will all find this in this presentation.
But first, imagine the following scenario. So there are like two engineers
discussing. I wrote code for the spacecraft. How do you know that it
works? I just know, or I
feel it. So NASA code, that is actually what happened in
NASA. So the code of the NASA spacecraft, it was
not tested. So in December 1998,
NASA launched the climate orbiter spacecraft
into the space. And this divide weighted almost half of
a ton and was sent to the Mars. It takes around half a year,
like six months, to go from Earth to Mars. And everything
was fine until September 1999, when all of
a sudden the contact with this device was lost.
What happened? So there was a big in the code. So the ground computer calculated
everything. In no metric system, they used pounds.
However, the orbiter used metric system, and there
was no conversion in between. So the speed of the orbiter was too
fast, and the orbiter passed too quickly through the atmosphere
of Mars and burned. So there was no people on board. So we can say,
okay, shit happens, it's just money. But you know what?
These NASA devices there are super expensive.
So this one costed more than 300
million of dollars. There's around 234
Americans working all their lives for these orbiters.
In case of Poland, it's even worse, you would need like 700 polish
people working all their lives to get this money. And that is
just because of the bug in the code. So how this conversation
should look like. So I wrote code for the spacecraft.
How do you know that it works? I wrote unit tests, but how do you
know that your test works?
But really. So we write code, and then
we write test to test the code. And then we write test to test the
test of the code, and write the tests of. To test the test of the
code. And it just doesn't make sense. So, if we are not
sure that our tests are good, so does testing
make any sense at all? And actually, yes, there is a method of
testing your tests without writing more tests. So imagine
we have the following code. So just the simplest possible code.
A plus b. So this could be a production code of, I don't know,
some calculator. So we write unit test. How do we check that our unit
test works? We could run test coverage. But what does
the code coverage check? So, according to common sense,
or according to the guru of the common sense, the Uncle Bob coverage
does not prove that you have tested every line. All it proves is that you
have executed every line. And that is a big difference.
If you think about it like this return
statement here, it would be perfectly covered with the test,
without any assertion. Just imagine the following test.
And we have a test like this, and we
have a 100% coverage, but we haven't tested anything. So we need
something better. We need something better than code coverage. And actually there's
something better. It was discovered by Richard Limpton
in 1971. So Richard Lipton, he asked like a
more fundamental question. He asked like,
why do we write tests? And he came to the conclusion that we
write tests to detect bugs. So think.
But it for a moment, if you are sure that your code does not have
any bugs, then there is no reason to write tests. So the
test is good when it catches bugs. So we could
reverse this strategy and introduce artificial bugs,
and check if our test detects these artificial bugs.
And that's what Richard Limton suggested in his paper.
He mentioned that if you want to know if a test suit has properly
checked some code, introduce a bug, how to do it in practice.
So let's go back to our example, simplest example possible. We have
our statement a plus b. So how can we introduce a
bug here? We could reverse the statement and change it to minus.
That is clear, a bug, because we wanted to have a sum, but we actually
have. We subtract the values. So what I did here actually I
created a mutation of the production code with an artificial
bug. And now we can check, okay, if our test suit fails on
this bug. If yes, our tests are good. If no,
our tests are useless. So you may ask, like, is it the whole idea
behind the mutation testing? And actually, yes, that's the whole
idea of the mutation testing. But before we go any further,
let's set up the terminology we
will use. So, this artificial big is called a mutation
operation. Code with the artificial bug is called a mutant.
When the test fails on a mutant, we say it killed
the mutant. However, if the test succeeds,
even if we introduce this big, even if create a mutant, we say that
the mutant survived. So, coming back to our example,
if the mutant is our mutant, a minus d
is killed, then our tests are good. If it survives,
our tests are bad. So in this case, like,
killing is good. So if all the mutants are killed,
our test suit is perfectly fine. Now, the first thing you can think about
is that, okay, but my code is much more complex than just adding
two numbers. And that is why we have a lot of different mutation operations.
And there are so many of them that we even put them into categories.
So, first thing, how you can mutate your production
code is to do some math changes. So we change some
plus to minus. We change multiply to divide
minus to plus. So we change all the math operations.
Second thing, we change the boundaries, so we had less than
a, less than b. So we changed it to less or equal than b.
We can also negate the statements, and we have some more complex
mutation operations, like remove if statements, remove method
calls, modify return statement, modify some constants,
and there are even more. So now what do we do? We have our production
code base, like the whole code base. We create a mutant,
so we use mutation operation to generate a mutant.
We actually create a lot of them because we have a lot of mutation operation.
Our code is big, so we create a lot of mutants. And now, if all
the mutants, if for every mutant, at least one tests
fails, we killed the mutant, so it's all good. However, if there
is at least one mutant that survived all our
test scenarios, it means that we didn't cover this code by
any test. So it is bad. So the next thing you can ask is like,
but do I need to change the code on my own? And luckily,
no. For Java, there is a very good library called pit.
And apart from being a good library, it has this great logo,
one of the best logo on my personal classification of logos
that goes, trust after Docker. This logo of
this bird is great. And you can use this tool, the speed mutation testing
tool. You can use it from command line, you can integrate it with maven
or gradle, or you can use it as plugin to intellij.
I actually always use it with my intellij. Just click ok, check if
my tests are good. So let's see our example again.
So we have our calculator method. We have our test,
which provides 100% coverage, but actually it tests
nothing. So how does it look like in
practice? So let's see a short demo. How to run
the speed mutation testing on this simple example. So what we're
going to do, we have just one class with the calculator.
This is exactly what we've seen on the slide. The simplest possible code. And we
have a unit test for this with our unit test provides 100%
coverage. But that's nothing. We can run the tests.
Obviously it passes. It will pass even though we had no production code.
We can even check what is the test coverage
of this. And yes, it's 100% class method
line. Everything is perfect. So now we need to improve this
process of checking our tests. So we will use the pit.
I already have the pit plugin installed into my intellij,
so I can run this edit configurations and
then add a configuration for a pit mutation testing
runner. So when I edit, I need to specify a few parameters.
I can give it a name, but doesn't matter much.
I can specify the target classes,
the source directory, the report
directory. So if you look closer, so it just specify where
should be the pit report generated. And that's basically it.
So if I click ok, I can run this mutation
testing framework and see the results in the results here,
it created two mutants automatically. And you see, they were
generated, but they were not killed, which is
obviously bad. So, coming back to the slides, so what we've
seen, this is the result of our pit mutation
testing. Obviously the mutant was generated because this
plus was changed to minus, but it survived, meaning our tests
are useless. How can we improve it? Obviously write some
better unit tests. In our case, we can change it to
the proper unit tests, like giving some ab
values given when we summit,
we should assert it to free. This looks like
a valid unit test. And this, if we run it again, it will result in
an output like this, meaning mutant was generated, but it
was killed. Perfect. Our unit tests are great. Now do I
need to read the console? So, I mean, the reading console is
not the great way of presenting the test results.
And luckily no pit provides a very nicely generated
HTML report and let's see how we can use it.
So let's continue with this demo with our corrected test. So if
we go to our intellij, we have corrected
our unit test and if we run
again our pit test runner,
we should see that, okay, mutants were killed, but apart from the
fact that it was killed at the end of the log, we can
see open report in a browser. So we can open here directly
open from intellij, the report that was generated in the browser and in the
browser we have a very nice report with the code
coverage according to mutation testing. So it's way
better code coverage than the standard code coverage. We can browse it by packages,
by classes and we can see what happened with our code.
So this is a perfectly well covered code according to mutation testing.
We can even see that two mutants were created out of this line
and all of them were killed. So this
is the output you are looking for. Okay. But the next thing, when I
first heard about mutation testing, I thought the
idea is great, I buy it. However,
it will never work for a bigger project because my project is way
bigger than just one calculator class, how it's possible that it works.
And what I tried here, what we actually tried in hazelcast was
to try it and use it on one of our plugins,
which is hazelcast Kubernetes plugin. It has around 5000
lines of code, so it's still small, but it's like reasonable
size. It has twelve classes, so not a very big code base,
but already something that is useful. So let's see in the demo how
to run the same pit framework
on the hazelcast Kubernetes plugin. If you go to the hazelcast
Kubernetes page, what we did is if
you would like to run it from the gradle or from the maven, you need
to add the dependencies to pit. So in
maven we added this with the profile.
So we have a pit test profile and it's enough to
add this part. And that's everything you need to change in your
project to actually automate it. So with this part we can see how
it works. From the command line we can clone this project. If we
clone this project, we can. Then let's code this.
And then if we open this directory
with the project we can run the command with pit test
with our profile. Pit test tests, mutation coverage this
command will generate for us the report. So actually
it takes some time for the pit to generate it. So maybe I will
not show it here,
but let's see how we did
it. So what we did with this Kubernetes plugin, we added
a GitHub action which runs this. Every time you
push to the master, this GitHub action is run.
And we run exactly the same command with the pit mutation coverage
in our GitHub action. But apart from that we also publish
this result to the GitHub pages. So we always have
in the GitHub pages our current result of the mutation test.
That is actually great. You can always go to this GitHub page and see the
results. And with this GitHub action we can see how it was run.
It's actually great because after every push to the master we
have pit results. We can have a look how it runs. So pit
coverage, it took actually 2 minutes and a half
for this GitHub action to execute all the tests,
mutation, everything. You can see that there is some mutant killed and
the results are automatically, the report is automatically
published to the GitHub pages. So we always have at the GitHub pages
current pit test coverage report. So we can open this and
see where are the things that are not
well covered with our tests. And if we look at the,
for example, this class looks like not well
covered. So we can see, okay, this line looks
not covered, right?
This looks like the whole method is actually not covered. We can see like
what? So if we change it to return now,
it's actually no test catches it. So it's really bad and
so on and so on. So we have like a code coverage,
but done way better than with the
standard code coverage tool. So I guess I already convinced you
to use it. I mean, it looks great. So the next question is like why
NASA Engineers didn't use it in 1998. So Richard
Limpton discovered this in 1971. The first like Java
mutation testing framework was called I guess Jepster,
and it was created in 2000 and the pit was
in 2012. So actually why
it took so many years from
the idea to go to something that you can use on
the code and you can use as a developer. And it happens that there
were two reasons why it was not used, widely used
for such a long time. And the first reason was the problem with equivalent
mutants. So let's look at this code. So this is a good code,
I mean it makes sense, this code, but now think
about it like if we mutate the second line,
this code is the same. Actually the semantic of changing
this mutant is the same. So we create a mutant, but no
test will kill this mutant. So we
have a false negative and
that is like problem. Actually, the pit didn't solve
this problem because this problem is not easy to solve. How to
eliminate these equivalent mutants. However, what Pete did is
that the mutation that are highly
probable to generate these equivalent mutants,
they are disabled by default. You can still enable them, but we
don't want to have false negatives
here. But there was also a second problem, why mutation testing was not widely used.
And that is because of the slow performance. Because think about it,
like from our code base, we create a lot of
mutants. I mean, we can change a lot of things, so it
results in a lot of mutants. And now we have a lot of
tests. So you can already guess the problem. So the problem
here is that we need to check every combination test and
it's super time consuming. So what pit actually did,
and it was quite smart. Before running this mutation
testing, they run the normal code coverage
and see which test covers this part.
So if we know that this test covered this code,
then we don't need to run all the combination. It's enough if
we run this test for the mutants related
to this code and it actually speeded up the
whole process very, very well. So we don't run all
tests on all mutants, we just run the test that may
kill the mutant for the given mutant.
Okay, that's cool. Actually, I guess I already convinced
you to use mutation testing. But you may ask, but what about my team?
They are lazy, they will not use it. And let me tell you a story
here. So I worked in the banking industry some time ago, and you
can imagine, like banking industry is like NASA, it's like
super important, big money. And our team was distributed
across three locations. One was in us, the other was in Krakow in
Poland, and the last one was in Hyderabad in India. And the story is that
each team was quite independent. So each team developed their own
part of the system, their own modules. And we
wanted to have 100% code coverage because we wanted to
have high quality. That was the most important, because banking industry
compliance, you need to have code well
covered. The system was constructed in a way that each module
had some main class. And the main class, I call it
main class paradox, because main class is difficult to test because you
cannot create a simple unit test for the main class. Because if
you create a unit test to run the main class, you're basically testing the whole
module, not only the main class. So there was, no matter how I
tried to tests it with unit test, I always could go to 95%
code coverage, never 100%. And then if you don't know how to do something.
If you see the problem, you see, okay, let's see how other teams do it.
And then I check the code from the team
from India. So I look at the code, I check all the code,
check all their tests. And what I found is that I
found, okay, for each test there was given when,
but there was no assertion. So they started to create
tests with no assertions because there was a requirement
for 100% code coverage and they needed to fulfill
it somehow. And I don't know, I don't know what to do with the people
who write tests without assertion. Should we laugh on
them or should we get angry? But the moral of the story is that the
test coverage threshold doesn't make any sense, because if you do it for
your project, people will try to work around it if
they need to. But you still want to have some cold coverage. So luckily there
is a better way. What you should do is put your pit framework into
your continuous integration or continuous delivery pipeline
and generate these reports of cold coverage. And once a week,
or once every other week, have a meeting and look at this report.
Not to have like a threshold, you need to have 100%
code coverage, but rather look at this HTML report,
because that is the way you
have less technical depth and improve the quality of your code.
So the last thing is like, but do people really use mutation testing?
Is it like widely used in an industry? And actually, yes, there are a lot
of companies. It is used in CERN, it was used in
the Norway voting system. And more and more companies are
introducing mutation testing to their pipelines because it
just improves the code quality. It's just better measurement than code
coverage. So test your tests because it
gives you the freedom, like you can easier refactor the test,
refactor the code. You can trust your, you know,
some shit can happen, or you can lose a lot of money,
or you can feel shame or had laughter or anger.
Thank you for listening to this presentation.