In the Universe of k6

Video size:

0:09 Miko Pawlikowski

Hello and welcome to Conf42Cast, your podcast from the neighboring galaxy. My name is Miko Pawlikowski and I'll be your guide today. And today I have two amazing guests, Nicole van der Hoeven, developer advocate at K6.io. Hello, Nicole.

0:27 Nicole van der Hoeven

Hi. Thanks for having us.

0:29 Miko Pawlikowski

And Simme Aronsson developer relationship lead at K6.io also. Very nice having you today Simme.

0:38 Simme Aronsson

Awesome being here.

0:40 Miko Pawlikowski

Thank you so much for finding the time. I really like the episodes when we have two people who actually know each other, that always helps with the dynamic. So we have got so much stuff to cover on my list. And I'm really excited for all the performance testing that you're going to tell me about today. But before we start with that, like the first thing that I actually noticed, when I was kind of looking at your online trace, you know, nothing is hidden anymore, is that you speak five languages and learning two more. And by "languages" I mean, actual human languages. How come? That's really awesome.

1:19 Nicole van der Hoeven

Thanks. I love to travel. And I really like talking to people. And I think that it's different when you talk to someone in their language. It always makes for a more genuine connection. It shows that you're willing to go the extra mile to meet them where they are.

1:37 Miko Pawlikowski

That's definitely the case and you seem to have gone an extra mile and then some with Esperanto, you're officially the only person I know that actually speaks that language.

1:47 Nicole van der Hoeven

I nearly wore my Esperanto shirt today too. Yes, I do. And you'll be surprised that I've actually used Esperanto quite a bit during traveling. Well, during my travels. It's been more useful than, let's say Dutch, which is way bigger than Esperanto. But I've spoken Esperanto in other countries, and I haven't really done that with Dutch.

2:10 Miko Pawlikowski

Wow. Okay, yeah, that does surprises me a bit. I personally have never heard anyone speak. Can you say something in Esperanto?

2:19 Nicole van der Hoeven

Yes, [speaking in Esperanto]? What do you want me to say?

2:26 Miko Pawlikowski

Say: 'performance testing is amazing'.

2:28 Nicole van der Hoeven

I would probably say something like [speaking in Esperanto].

2:37 Miko Pawlikowski

Okay, so it kind of sounds vaguely Spanish to me for a complete like on that, but it's really awesome. Okay, well. And then obviously, you probably going to be, when you spoke Esperanto, was it to other people who specifically learned that language to kind of talk to strangers? Or was it because it's kind of similar to other languages and people could guess what it meant?

3:04 Nicole van der Hoeven

I originally learned it because I read this study that said that there were two groups of students and one group learned Esperanto for two months, and then French for four. And then the other group learned French for six months. And the first group scored better in almost every aspect of French, not Esperanto. I wanted to learn French. And you know, several years later, I speak Esperanto and only a little French. So, I kind of got waylaid on that.

3:39 Miko Pawlikowski

That's fair enough. Okay, that's a really awesome story. But for all the other kind of posts on your LinkedIn, you know, timeline, it seems to do much with performance testing. And I saw that you're actually giving a webinar later this week intro to load testing with Grafana and K6, and I really want to go into what Grafana Labs are doing, because a lot of that is very awesome, and K6. But before we get into that, so you seem to be a little bit obsessed with the performance testing. Is that an accurate way of saying that? Or is it a bit of an exaggeration?

4:15 Nicole van der Hoeven

No, that's probably accurate. I do really love performance testing. I think it's part of my personality to always want to optimize and be more efficient, not just with the software that I use or the software companies that I work for. But just in general. I'm a bit of a min maxer.

4:39 Miko Pawlikowski

That's fair enough. I'm saying all of that, but you Simme, you seem to have a similar trail really. I saw a little bit of SRE into that. And then I saw that you previously had your own freelance shop and then you moved on to K6. Is that accurate?

4:55 Simme Aronsson

Yeah. That's accurate.

4:55 Miko Pawlikowski

And also there's this common theme of the open source. It's also something I want to touch upon a bit later. But before we do that, so what's performance test really? Why is it something that people should be doing? Who should be doing it? And how do you actually define that?

5:13 Simme Aronsson

Okay, so performance testing really is, or performance engineering I would say, really is taking the art of or taking the practice of assessing your system, and what kind of throughput or what kind of performance you can expect to get from it. And making that into something a little bit more along the lines of what we do in engineering in general, I would say. So you're trying to systemize it and define practices and how to make it, you know, fit into the everyday developer workflow, or the tester workflow. So how we do it is usually by doing load testing. You can do it in many different ways, you can do profiling or other types of testing as well. But we usually do it as load testing. And for those who are not familiar with that, it usually means that we create a set of virtual users, so to speak, that will try to access some kind of application resource in a very fast pace, or in a defined pace. So you would say, for instance, that, okay, you have this cool Conf42 page that you want to sell tickets to your conference from. And you want to make sure that whenever you release a smashing SRE conference, for instance, you are able to withstand the pressure that all of the ruckus all over Hacker News will create as soon as you release the news, right? So if you expect, for instance, that you will sell 10,000 tickets the first hour, then you want to test for that particular circumstance. So then you'll use performance testing, or load testing, to be more specific, to simulate all those 10,000 users, navigating through your webpage, clicking the Add to Shopping Cart button, checking out everything and make sure that your system is able to withstand that kind of pressure.

7:08 Miko Pawlikowski

Definitely. And when you kind of talking about that it really draws parallels with, I spend a lot of time walking around talking about chaos engineering. And I think one of the interesting aspects is that you're basically spending more of your time on the runtime part of your software, rather than writing the code, right? As opposed to unit tests, when you kind of verify some logic, then you're actually more interested into what happens at the runtime when there's real hardware thrown behind it and are real users and real patterns and all of that, that kind of more closely resembles the real life. So I guess my question is, like, with everything going to the cloud right now, and the solution is seeming to be for every question right now, just make it microservices and put it into Cloud infinite scaling, web scale. Is performance testing, becoming obsolete by any chance? Or is it transforming? Is it becoming more crucial? How is it affecting it?

8:12 Nicole van der Hoeven

I think it's it's actually the contrary. I think the proliferation of microservices as opposed to a monolithic type architecture means that it's even more important that you performance test your different services, because I mean, microservices add a lot of advantages compared to monoliths, but they also add complexity. And that's a problem, because it's significantly easier to performance test just one large thing, rather than smaller services, because then you start to have to look at interactions between them. And you start needing some sort of proxy, some intelligent proxy that can redirect and manage the flow of traffic between them as well. That's not really something that you have to think about with a monolith. So I think if anything, it's getting more important. But perhaps the tools or the techniques that we use to performance test are changing as well.

9:11 Simme Aronsson

But I totally get a why people get, they might get the impression that it's becoming obsolete. Because I mean, you are probably in that case, focusing on the risk of saturating the input and output to the actual user endpoint that you will be directly accessing as a user. And while that is true, you might be able with the cloud and with microservices to split that traffic up and making it less of a problem with infinite scaling, as you say, or infinite scaling in citation marks, I guess. Yeah, but as you split them up, as Nicole says, into multiple different services, you also get multiple different processes, possibly running on different machines or even in different geographical locations. And that introduces a lot of latency. That's kind of part of the design, that latency will be introduced. Given that you will have to do some kind of requests between these services and have to build up this mesh of intelligent communication between these services. So while you might not need to test as much at the user API, or the user endpoint that you expose to the public Internet, you might very well end up with different kinds of jokes or bottlenecks that you haven't had previously in terms of communication between services.

10:35 Miko Pawlikowski

And that always brings me the memory of Istio project, I don't know if you followed that when they designed all this microservices, and then there were like, 'Actually, a monolith isn't that bad. Let's go back to that'. And they moved it back on. A famous example for everybody in the Kubernetes sphere. So you work and talk and present to a lot of people. What would you say would be the answer to the following question, is everybody who should be doing performance testing? Because with things like unit tests, there's enough social and peer pressure right now that if you're not writing, then you probably should look for a different job, maybe a baker? Is it the same with performance testing? Are we still not at this stage? Or are we no longer at the stage where this is something that's expected from people? If you were to like, say, like, what's the percentage of people that you interact with who actually do it properly? And what's the percentage that should be doing and aren't doing that?

11:36 Nicole van der Hoeven

I happen to think that everybody should be thinking about performance. I am a performance tester. But I also don't think that it's a performance testers' responsibility to make sure, I guess, a better way of saying it is that it's not a performance testers' sole responsibility. And they're also not the only persons who should be concerned with performance. Because it really is a team thing. Even as a developer, even if that's your primary job, you should be thinking about performance, you should be running some sort of performance test whether you're going to be running the full peak load tests, I mean, you could as well, but even at the unit testing level, there are unit tests for performance that you could be running. So I think from that perspective, yes, everybody should be doing some sort of performance.

12:33 Simme Aronsson

And I just want to add to that, that I think that, that's actually one of our primary missions at K6, I would say. We want to democratize the whole idea of doing performance testing. I mean, it's for so long, it's just been a tester practice that have required you to learn all these fancy gooeys, that you need to point and click into to create these projects and actually build your load test up. But that's not really necessary as we see it. And we actually spent quite a lot of time to try to make it fit into developer workflows as well, giving you the possibility to actually do performance testing using code and maybe even as part of your CI or your you know, checking in your performance test using it. And that has that has not not been the case previously. So I would say that more and more people are able to, or get the chance to approach performance testing. But not all are actually doing it. I also have a little bit different approach to it, because I think some people actually do performance testing that might not need to. I know, for instance, that both Nicole and I have our private portfolio pages where we post things that we do around the internet, I would say that mine at least, Nichols probably has 10 times more traffic, but mine has like 10 to 20 visitors every month or something like that. I don't really need to low test that. I know Nicole does for her site.

14:02 Nicole van der Hoeven

Yeah, I totally do.

14:04 Simme Aronsson

But I know for sure that GitHub Pages service will handle 10 to 20 people visiting each month. So while everyone should start to think about performance, I don't think that everyone needs to do load tests. So you kinda need to tailor your efforts to the project that you're actually working on. Let's say that you have an actual user base and that you have concurrent connections, you should start to think about getting into performance testing to some degree at least.

14:34 Nicole van der Hoeven

I think it's also worth talking about the difference between performance testing and load testing, which Simme alluded to. I think that people conflate the two. And I totally agree that not everyone needs to do load testing. But I do think everyone should be thinking about performance. And the difference is that there are many things, many activities that can count as performance testing, you know, running your page through Lighthouse. Lighthouse is already included in DevTools now. So, it's free, and it's pretty easy to just run it. That counts as a performance test. It's not a load test. But I think performance testing should be done by anyone who cares about the experience of people visiting your site. And even if Simma doesn't do load tests, I'm sure that he has also chosen a platform that is performant. So I know that you use Hugo, for example. And part of the reason that you use that is because the experience of a user is way better. That's thinking about it from the point of view of performance as well, even if it's not load testing.

15:48 Miko Pawlikowski

Isn't Hugo a static page generator?

15:51 Nicole van der Hoeven

Yes.

15:52 Miko Pawlikowski

That's old school. Let's pre-generate everything. No JavaScript. So one thing that did make me think about is at first, Nicole, you mentioned the keyword performance tester as a kind of job title. And then what Simme was describing was more like, making it an element of different job titles, job description. So we just say that it's the case that it was previously, you know, this dedicated people who would do that, and you know, they would have this long beard potentially, and their tricks. And now it's basically becoming more accessible and more, like you said, democratized. And it's becoming part of everybody's Toolkit, including, you know, the developers, SREs and everybody else involved?

16:40 Simme Aronsson

Well, I come from a developer background, right? And, and I know that, or I think that Nicole is probably one of the best performance testers that I've met. And I don't see any long beard on her. So I assume that it's not only people in sellers, with neck beards that do performance testing, right? But with that said, I do think there's still a place for people that are specialized at performance testing. But I think their role will be more of a strategic or coaching capacity, rather than actually being the ones that have to do all the actual load tests themselves. Just as with testing, right, when we, when we do cross functional teams, and we involve the testers in the teams, and have them be part of the team, we should probably do that with performance, as well, as part of your definition of done or your deliverables and your team, you should probably do some performance testing, just as you do unit testing, integration testing, UI testing, and things like that. So I definitely think that it's becoming more of a task or a practice than a role. But I still think the coaches for that practice, have a very important role to play in the organization.

17:52 Miko Pawlikowski

That's fair enough. I'm not gonna argue with that. Instead, I would like to shift gears a little bit and go to some of the talks today. So in particular, one that I really liked was the 'Schrödinger's Pokemon'. And, you know, I'm obviously biased for chaos engineering. But I was curious, where did that idea come from? And what would you say is the overlap in general between the chaos engineering and chaos testing in general, and performance testing from the perspective of someone who's coming from the performance background?

18:25 Nicole van der Hoeven

Sure, I think Simme actually had a lot to do with my introduction to chaos engineering. I think that at its heart, it is very compatible with performance testing, there are a lot of similarities, there are some differences. The difference is mainly I think, a slight attitude shift. In any sort of testing, at least traditional testing, the idea is to prevent failures or errors, but with chaos engineering, that is changed a little bit, because the idea is not to prevent them, but to reproduce them and accept as a given the fact that applications are going to fail. And perhaps that also has to do with the realities around, you know, microservices and event-driven architecture. The architectures of our systems are becoming more and more complicated. So it's becoming more and more difficult to unequivocably, you know, try to prevent a failure. That's just not realistic anymore. So I think chaos engineering comes from that need to react to what actually is. But the similarity is that they're still both very focused towards making the application or the system hardier, more scalable, more performant, more reliable. I think that the lines are blurring a little bit between performance testing or performance engineering, and also SRE, Site Reliability Engineering. I think that there are a lot of skillsets that are now overlapping.

20:02 Miko Pawlikowski

Definitely. So for everybody who hasn't seen, can you explain why it's a Schrödinger's Pokemon situation?

20:09 Nicole van der Hoeven

Sure. So, Erwin Schrödinger is a physicist. And he meant to say something about, he has this famous thought experiment where he puts a cat into a box. And it's a thought experiment. So the idea is think about what would happen to the cat if you introduced it to some sort of poison. Now, I'm sure he has other things to say, in the world of quantum physics. I'm not a physicist. So I just repurposed that to performance testing. And I think it is so true that when you're running these tests, the tests are almost worthless unless you know how the application is behaving. Otherwise, you're just subjecting the Pokemon in a box to a lot of random events, and you don't know how it's actually doing inside there. So the reason that I thought of that thought experiment is that K6, which is a company that I work for, was recently acquired by Grafana Labs, which is a company that has many, many projects, and a lot of them have to do with observability. And I was surprised by the opinion of some people that this was a surprise, or they weren't sure how it fit together. And I thought it absolutely fits together, because you need observability for performance testing, how do you know what's happening otherwise? So there's a huge overlap there. Because if you don't know what's happening inside the box that your Pokemon or your application is in, then how will you know if it's alive or dead?

21:49 Miko Pawlikowski

Yeah, I'm pretty sure that if Schrödinger was coming up with that today, he would have picked a Pokemon that's both dead and alive at the same time. Makes perfect sense.

21:59 Nicole van der Hoeven

Yeah, the Pokemon part is just because I actually really love Pokemon.

22:04 Miko Pawlikowski

Alright, okay, that's fair enough. And I really can kind of relate to the idea of Pokemon sitting in the box that's firmly closed, I like that. And also, I just wanted to mention, like the cubed in conversation that you had with Calvin, that's such a small world. Calvin used to intern in my team a few years back, I was like, 'Oh, yeah, this guy is going places'. That's really cool.

22:28 Nicole van der Hoeven

That's so funny. Yeah, he's brilliant. And I really wanted to have him on on our livestream, which is 'K6 office hours'. We have it every weekend, we just talk about different ways that people use K6. And Calvin had an interesting use, because he use it in his academic thesis, which is not something that we see a lot. But he has a good head on his shoulders and it's not so academic that it's infeasible. And he created this thing called 'cube dim', which is essentially a reverse proxy, that kind of intelligently manages traffic. So he applies concepts like brownout theory, and dimming to proactively stop requests from being passed on to other microservices, depending on the state of the system.

23:20 Miko Pawlikowski

Yeah. And the response time and everything. Yeah. Good conversation, and a shoutout to Callvin. Okay. So I think at this stage, we really need to cover what K6 really is and what it does. We know that you've worked there, we know that it has something to do with load testing and performance testing. We know that Calvin used that in his thesis. But what is it really? And why is it so cool?

23:47 Unknown Speaker

So K6 is mainly an open source tool, we've created K6 OSS, which is the main part of the actual offering that we have, that is used to create a load against the system. So what it does, basically, is that it allows you to, using JavaScript, create a series of logical. K6 is a load testing tool. It's open source and available for anyone to use. We also have a cloud offering that you can use, that takes care of some of the management for you. But mainly, it's an open source tool. It's used to create and apply load to your system. So you can use that to generate a lot of virtual users and have them do requests against their system. And it does that by utilizing a very, very performant Go backend that we expose to this JavaScript layer that the users may create their tests in. And so you build a test using JavaScript, you define what the circumstances should be for your test, and then you run it in this OSS backend that we've made available. And it will spin up these users and have them execute whatever you put them to. And if you use, for instance, the K6 operator or cases cloud, you may also specify how many load generators, how many cases instances you want to have, and how you want to distribute your users across those instances.

25:18 Miko Pawlikowski

All right. I love to find that it's open source. Curious why you picked JavaScript as the kind of user frontend language. But it sounds like it's basically a step up from tools like Apache Bench and Work and Locust and whatnot that makes it just easier to automate generating all of the traffic.

25:38 Simme Aronsson

Definitely, we were thinking like, 'Okay, so what language should we use as the user API or the user layer?' Or the scripting layer, if you will, and our previous product load impact V3, was written with Lua as the scripting layer. But then we thought at some point, 'Okay, so what is the most important characteristic of the scripting interface that users should be exposed to?' And we came to the conclusion that it's commodity. You want as many as possible to be somewhat familiar or somewhat, you know, experienced in whatever scripting language you use. And Lua, while very cool, definitely isn't that language. So what language would make better sense than JavaScript? I think almost every developer I've met has some kind of experience with JavaScript, either you love it or you hate it, but you've at least been exposed to it at some point. Which makes the learning curve a lot less steep than if we were to go for something completely different.

26:41 Miko Pawlikowski

Yeah, until you need to do a loop. Well, that's a good point. And I guess it's completely independent of how your software is actually written, right? It's just a script to test themselves. And it's probably a solid choice. And yeah, I can see why Lua wouldn't be, like, popular enough.

27:06 Nicole van der Hoeven

I also wanted to say that for testers, it's also really good because most automation testers, even if you're, you know, doing functional testing, have had some sort of interaction with JavaScript as well, because tools like Selenium, WebDriver, or Puppeteer, or Cypress, those are all written in JavaScript. And so it's a lot easier even for a tester to get started. And the whole idea is that we're trying to bring load testing from this complicated, really monolithic tool that requires licenses and a full environment and a different language. And there are load testing tools that have their own scripting language, to something that just seamlessly fits into your workflow. A language you already use, an IDE you already use. It just made sense to us on a lot of levels.

28:05 Simme Aronsson

Yeah, some popular old school tools even use C as their scripting language.

28:11 Nicole van der Hoeven

Hm, which one are you talking about?

28:13 Simme Aronsson

There are a lot of different alternatives. And I think, out of the ones that I've seen, JavaScript is probably, at least for me, the most reasonable or the easiest one to digest.

28:23 Miko Pawlikowski

Yeah, and I can see kind of classics XKCD reference with it's compiling upload to my script for load testing being a problem. For everybody who wants to test it out, I guess they go to github.com/grafana/K6 now, or is that the best place to start?

28:49 Simme Aronsson

I would say the best place to start is probably to go to just K6.io and just click the docs section. And you will get some pretty neat guides on how to get started. If you prefer to start from GitHub, by all means, but I think you'll have an easier time getting started if you start in the documentation.

29:10 Miko Pawlikowski

One of those unique projects that actually have good docks while being open source, that's extra points for you. And for everybody who's been confused. So you used to be called Load Impact, right? And then you rebranded as K6, because I'm guessing that was your flagship project. And now you're part of Grafana Lab, but you're still operating more or less independently. Is that an accurate description? So what changed?

29:40 Nicole van der Hoeven

It actually started as Gator hole. I think that was the first name of the company. And they were making an MMO RPG. I am sad that I wasn't around for those days.

29:54 Simme Aronsson

I think it was called the MMO RPG and we've actually jokingly told the CEO of Grafana that, 'Did you know that you also bought an awesome IP for an MMO when you acquired K6?'

30:09 Nicole van der Hoeven

That's really what they bought us for.

30:12 Simme Aronsson

That's just a bonus.

30:14 Miko Pawlikowski

That makes sense. Okay, so apart from extra IP, what changes for you like day to day being part of Grafana Labs?

30:23 Nicole van der Hoeven

We have a people ops team. That's one of the main things, I think we get a lot more support for a lot of functions that were sort of split. We don't have HR in K6, where a team of 30+ people. And so there's no HR, there's no hiring team. It's really just the hiring managers and the colleagues really who do the interviews and do the screening. So I think, at least in the short term, there's not much that's changing for us in terms of direction, or really anything else, even the team structure, we're still operating pretty much as an independent unit. And we have actually already been using, we had been using Grafana internally, even before the acquisition, and the people at Grafana Labs had been using K6. So using each other's products was not something that started as a result of the acquisition. That's actually one of the reasons why this acquisition made so much sense, because we were already fans of each other's products. But in the long term, I'm sure that the direction of both products are going to change and move towards each other. But I think that's also a pretty natural progression. We've already talked about how observability and performance testing go hand in hand. So that is the direction that the market is going as well. So I think it makes a lot of sense.

31:57 Simme Aronsson

Definitely. And it also doesn't hurt, the fact that we've gone from having 30 awesome colleagues to collaborate with to suddenly have over 400 colleagues to collaborate with. So it's definitely a huge boost in terms of the collective knowledge or the collective possibilities that we have, as opposed to being just this tiny player with 30 employees.

32:22 Miko Pawlikowski

Are they all playing your game now? All the 300 colleagues playing your game now?

32:28 Simme Aronsson

We play so many games now, which is kind of awesome. So if you don't enjoy the current game, there's always other games to join as well. So I think it's really cool that we have that opportunity to be involved in each other's projects, and, you know, share knowledge and share experience across those project boundaries. So now they're not all playing our game. But we get to participate in their games as well.

32:55 Miko Pawlikowski

That's awesome. So, I wanted to kind of use that opportunity to maybe introduce some of our audience to some of the products that are also open source and are also awesome, but are not called Grafana. So they might not know about and so could you do like a quick tour of force of things like Loki and Tempo and Tonka and you know what makes them awesome, and how they can be useful to our listeners?

33:26 Simme Aronsson

Sure, I mean, I'm personally super excited about Tempo, I think is a really great product. And what it does is that it serves as a backend for distributed traces. So you'll be able to do distributed tracing using Jagger, or open telemetry or what have you, and then have that visualized as part of your Grafana UI as well. So you'll be able to drill down in these traces and look at values that are produced as part of your trace. So it really brings it all together in such a neat way that you're able to get all the metrics out of your system or out of K6 and visualize those, and also correlate them with whatever traces you have been generated as you do these requests. So you get the full experience of having high visibility, high observability into your system, which I think is just smashing. And it's also open source just as the other projects, as you said.

34:30 Miko Pawlikowski

Certainly. So you said the backend, does it include the storage layer, or does it still need some kind of database for the actual storage behind it?

34:40 Simme Aronsson

Well, you actually write your traces directly to Tempo. So I think it actually uses, I'm not really sure about the internals, I usually use the Grafana cloud offering. But you don't need to have anything separately set up you offload them to Tempo, and then you add that as a data source to your Grafana instance to be able to inspect it and pass it.

35:09 Miko Pawlikowski

That makes sense. A nice plug for Grafana Cloud, starting at 9.99. Just kidding.

35:16 Simme Aronsson

Starting at free forever.

35:18 Nicole van der Hoeven

Yeah. For us, it's free. So like, why wouldn't we use it?

35:25 Miko Pawlikowski

Makes sense. Okay. Yeah, Tempo really sounds awesome.

35:29 Nicole van der Hoeven

I actually made a video recently, it's called "a testers overview of Grafana Labs open source project", because I was also a little confused. I knew about Grafana, just Grafana as the primary project, but I wasn't too sure about the other projects. And I guess I should preface this by saying that we have just joined Grafana, like two months ago, so don't take it as a company line, this is how I make sense of it. They have the three so-called pillars of observability, right, and what Simme was talking about Tempo falls into the traces. And the other two are logs and metrics. So for logs, Grafana have Loki, which is a log aggregation system. And then for metrics, they actually have a bunch of different projects. There's a kind of separated into two, which is like the pull-based monitoring and the push-based monitoring. And there's Graphite, on one end, there's graphite, and on the other, there's Prometheus. So they're both time series databases, but they also are able to their actual stores of the data. And then they're both not really made for multi-tenant system. So if you want to scale that out, for Graphite, there's Metric Tank and from Prometheus, there's Cortex. And the one that I always forget is Grafana Tankah, which is a configuration utility for Kubernetes cluster. That's the one that I've had the least experience with. I've actually never used it.

37:04 Miko Pawlikowski

Yeah, I was just having a quick browse of what it's actually doing. Looks like it's replacing Jason, well Yamo, with some more custom layer. All right. And Loki is kind of like Prometheus, but for logs, is that what it is? Well, for me personally Loki now makes me think of a crocodile because of Disney+ series. Or was it an alligator?

37:30 Nicole van der Hoeven

You don't think of Norse God?

37:34 Miko Pawlikowski

What problem does it really solve, Loki?

37:37 Simme Aronsson

It's just as Nicole says that, it really allows you to get even more observability into your system, right? Because you usually have logs and you have them distributed across your applications or your services. And you'll have to manually digest or, you know, expose them to actually be able to browse them, but you want to bring all of that together into preferably Grafana. And make sense of it all at the same time. So say, for instance, that you have a metric that is subpar to what you're expecting, then you can add Loki to be able to add logs as annotations to that metric as well. So you'll be able to, you know, drill down into that log and and check, 'Okay, so what was happening? Why is it slow?' and accompany that with Tempo and the traces that you get. So, 'Okay, these were the values that were put in. What effect can that have on whatever results we were getting?' So it really ties everything together in a nice way. But for to answer the more specific question about Loki, it's a good way to aggregate logs from multiple sources and have them available in one place.

38:42 Miko Pawlikowski

So does it replace something like Splunk or Humio as the log aggregator? Or is it specifically trying to attach relevant logs to metrics for that purpose? Do you still need to have like the central log aggregation outside of Loki, if you use Loki?

38:58 Simme Aronsson

I have not used, maybe Nicole knows better, but I've not used it extensively yet. But my impression is that you don't need to have anything else you can just push it directly.

39:09 Nicole van der Hoeven

Yeah, I don't think you need anything else. You could still mean all of Grafana's projects are very composable. So you can use them with other tools as well. But yeah, I think you don't need to, you can just use Loki and have it sent to Grafana. They basically turn, I think it turns logs into metrics kind of, because it uses the same label sets that Prometheus does. So that's one of the reasons why they work so well together.

39:39 Miko Pawlikowski

So I know that a lot of people just kind of pick up this tools one or another when they need it. And that's how they get started with the, let's call it, Grafana ecosystem. What would you say would be like the best way to start with that? And here's where you plug the Grafana Cloud.

39:54 Nicole van der Hoeven

I don't know if Grafana Cloud would be the easiest way to start and really like even with K6, if you're just starting, I mean, depending on your proclivities or your experience, your interest, I guess, I might actually just say, 'Go and install K6 OSS'. And then the same way, I'd say, 'Install Grafana for free locally', because that's a good way to start. And if you're on a Mac, like I am, like, for K6, that's brew, install K6, it's, you know, it's just really easy to do that. I'd say start with the open source alternatives, and then see if he even need to go to the cloud.

40:32 Simme Aronsson

I totally agree. And I mean, just downloading the Docker compose file and having the whole Grafana stack being set up on your system as Docker containers, that's probably the easiest way to get started. Start with that, have a look, see what products you need, and to what extent, and then you can consider the cloud alternative, if that makes sense for you. But with that said, if you don't want to do all of that, you get a free, there is a free tier of Grafana cloud that you can use that include quite a lot to be a free tier. So you can get started and go quite a long way with just that. So if you don't want to set it up locally, you want it to be in the cloud for some reason. Or you just want to use a cloud service, go for Grafana Cloud by all means. You can try it for free, so you don't have to make any upfront investment for it. And then if you like it and need more than there are several plans to choose from.

41:20 Miko Pawlikowski

Okay, and that sounds like a really good segue into the closing notes. So you know how to start, you know, what the performance testing and load testing is about. You know, where to find our guests, Nicole and Simon K6.io, potentially, is that LinkedIn or Twitter to reach out directly to you?

41:43 Nicole van der Hoeven

Twitter, probably.

41:45 Miko Pawlikowski

Twitter, probably. Simme?

41:47 Simme Aronsson

Yeah, same. And if you want to find me on Twitter, you can find me at 0X12B on Twitter.

41:53 Miko Pawlikowski

0X12B, there we go.

41:55 Simme Aronsson

And Nicole, where can we find you?

41:57 Nicole van der Hoeven

Oh, no, I have to spell my name, @N_VANDERHOEVEN.

42:04 Miko Pawlikowski

Don't worry, people will be able to pause and rewind that, so it should be just fine. Alright, well, so before I let you go, I let you off the hook, I just wanted to squeeze one last bit of information out of you. And the question is the following. If you were to pick a single highest return on investment thing that you did for yourself and your career, and it can be anything, from picking up yoga to reading a book to going on the retreat, or learn a new skill, what would you pick, if you just could pick one single thing?

42:44 Simme Aronsson

For me personally, it's been if not as general as exercise, then maybe going out into the woods, just doing something physical that is totally non-related to what you do professionally. And just find that time to unwind and let your thoughts marinate, so to speak, while you do something else. It's really been a high return on investment for me. A lot of great ideas come from being able to completely unwind and recharge and then not think about your professional work for sure.

43:20 Miko Pawlikowski

So marinated thougths are the best, good for your microbiome. About you, Nicole?

43:25 Nicole van der Hoeven

If I could only give one piece of advice for anyone who is a developer or tester, or SRE or really any sort of knowledge worker, it's take notes. Learn how to take good notes, and develop a system, a personal knowledge management system around those notes. So you're not just taking notes and then forgetting them. I personally use the zettelkasten methodology that was pioneered by Nicholas Lumen, who's a German I believe, but you don't have to do anything super extensive. I think it provides so much value to have a second brain. You dump things into it that you are so sure you're going to remember years from now, but you will not. You will completely forget what you were working on. The things that you think are very obvious are not going to be obvious in a few years. Technology, especially if we're working in technology, in tech, it's just moving so quickly all the time. There's so many different languages and frameworks and concepts we can't possibly keep up and that's where you need to find something that you're going to stick with. And at its simplest form, use markdown, use something that you can store your notes in, in plain text, use something digital. I understand analog is really awesome and I have my own fair share of fountain pens, but trust me digital is the way to go. Because you can search it. And then also having a system where you're reviewing the things that matter to you, and you're consolidating your thoughts. So you're not just parroting what other people say, but you're actually thinking about how it fits into your view of the world of technology. That's essential. And it seems like such a non-technical answer, you know, take notes. It just, it's simple. And we all think we know how to do it, but we don't. And let me just recommend how to take smart notes by songket errands is a really good place to start.

45:35 Simme Aronsson

I really like your usual analogy, when it comes to note-taking, the one with how to performance test LDAP or Active Directory services, where you were like, 'I have no idea how to do this. Okay, but turns out, I've actually done it in the past and that I have a note explaining exactly how to do it'. And that's just such a time saver when you're facing problems that you might have encountered before. But given our limited capacity to store this information, you don't remember it from the top of your head.

46:08 Nicole van der Hoeven

Yeah, apparently I've tested it before. And I use j meter. And I have everything written down there. Like how how to do a thread bind and J meter and why thread bind is there and actions and thread dot and by that all these things. And really, if you had asked me, I would have said I've never tested LDAP I don't even really know what that is.

46:27 Miko Pawlikowski

That sounds like really good advice. Well, that is until NeuraLink develops the direct brain to computer connection and you can plug in directly into your website. But until then, that's solid advice. Thank you both. Nicole van der Hoeven and Simme Aronsson, thank you for your time and see you next time.

46:49 Nicole van der Hoeven

Thank you for having us.

46:52 Miko Pawlikowski

It was a treat. Thanks, guys.