My name is Moshe Zadka. My website is cobraism.com, where you can find every way of getting in touch with me known to humankind. And today I want to talk to you about Python sidecars. I do want to start with an acknowledgement of country. I live in Belmont in the San Francisco Bay Area peninsula, the ancestral homeland of the two Sholoni people. I want to start with a quick Kubernetes refresher, make sure that we're on the same page. Kubernetes is pretty big, so this is definitely not a comprehensive Kubernetes review. So let's start with nodes and containers. A pod is a basic execution unit. In Kubernetes that's what we execute. It's usually not what we configure and that's going to be really important both of these things later. A pod is made of n containers. A pod can have more than one container. They're all going to be in the same pod. They do share a network space, which means for example one twenty seven zero zero one is the same ip for all of them. They might share the process namespace. You can configure it on and off and we'll talk a little bit, but this later they can share volume mounts, so you can mount the same volume into both of them and they will share that part of the file system, but they never share the overall file system. So this is like the little bit of things that are going to be important later as we are talking about how to operate your sidecars. The other thing that's important in a pod is that readiness. Readiness is kind of a weird thing that is basically the criterion. Is the pod good? Why do you need to know if the pod is good? We'll talk about this later. This will become very important later. The usual way to determine if the pod is good is do TCP or HTTP check, which means you are sending in a TCP checks. If it connects and does a TCP hand check, it's successful. The more sophisticated checks that Kubernetes can configure is the HTTP check, which will fail on any status that is not two xx. You can also configure command checks. Basically there's a whole raft of issues around them. Usually I would not recommend them. And we'll talk about some of the ways that writing sidecars can alleviate the need to write command checks in Kubernetes. The next thing that I want to make sure that we're all on the same page in is deployed and services. There's many other things that you would need a sidecars for that are not deployments and services. But since these are the most common, let's focus on them for this talk. So a deployment is kind of a weird Kubernetes object because it's really just something that will automatically configure a replica set and a service. So if you understand those two, you also understand what deployment is. A replica set is a collection of identityless pods. And what I mean by identityless, there's no sense in this insurance things. Pod is the fifth pod in the set. If it's six pods, there's six pods. But if two die, it will bring up two other nodes, but it doesn't which one is like the true continuation of the next pod. There is no sense that this would make a sensible question. A service is something that can route to good pods. There's many ways to route and I will not go into them because for the most part they're not going to affect our talk. But it's very important because if you remember earlier, I said that radiance check is deciding what's a good pod. The reason you need to know if a pod is good or not is these service can know whether to route to it or not. And again, all of these things are a very important context because they will be important later. Side cars, not an atomic thing in Kubernetes there is no point where you will write in your Kubernetes YaMl configuration sidecar. You can if you call something a sidecar, but that's not a concept, that's not a type in Kubernetes, but it is a pattern. And specifically in Kubernetes, often when we say a pattern, what we mean is it's a shape that a YAml file takes. And when the YaMl file takes that shape, we'll call the resulting thing a side cars. What does a sidecar pattern look like? It looks like a container, a pod that has an extra container. And usually it just needs a second container because usually you have a main container that does the things that you are doing with the pod, right? If you think in terms of like a web application or a web API, this is a thing that actually runs a web server, right? If you think in terms of a queue consumer, this is a thing that consumes a queue and does something with it, nodes processing and takes it away. The idea of the sidecar is that it takes care of the rest. And obviously this is a pattern. So all these definitions of sound fuzzy and to be very honest, it's kind of not really a meaningful debate to say whether a specific thing is a sidecar or not. What's important is to know that this pattern is a pattern that will be really useful when you're designing a kubernetes installation. So you say, oh, well, what could be a useful sidecar here, right? There's no reason to get hung up on the definition of whether it's a true sidecar or not. So here's like an example of a sidecar. The main container writes a web application, and then these application wants to cache to files. Now, usually it's kind of awkward to also have the garbage collector sitting in the same process, right? Because web servers run in a specific framework that's got to manage all of these things starting to shove like an extra thread or spawning a process and then having to manage what happens if process dies. That's why we have kubernetes for, right? Like the whole point of kubernetes manages that kind of stuff, right? If you wanted to manage that, you wouldn't be using kubernetes in the first place. So what you can do is you can have a sidekick container and that looks at files that are like, say too old and removes them. So it basically functions like the garbage collection thread or process. But now Kubernetes is managing it. It's managing a different container. And one of the things that is a different container, it is built separately, right? You build those containers into separate like CI pipelines or whatever it is that you use to build containers. And so you don't have to use the same base image, you don't have to use the same platform, you don't have to use the same language. So why I thought has useful. So first of all, you can separate the resource limits, meaning you can make sure that your web application can take only so much memory and your sidecars does. A garbage collection can take so much memory and you know how much memory you allocate to each. They can't kind of go and steal from the other one's memory. And that's very useful when you want to do very careful capacity planning. At some point you will need to do. They're also legible to Kubernetes dashboards, right? If you put both of these things in the same container and they die and they get restarted, you'll have to also make sure that you keep track of that somehow to make it legible. If you put them in separate containers, Kubernetes knows if a container died, Kubernetes knows to separate the container's logs and so on. So that's very useful that you can, obviously there's like the native Kubernetes dashboard and quite potentially like whatever cloud provider has more stuff around kubernetes. So that's why I don't say the Kubernetes dashboard, it's general Kubernetes dashboards, right? It might be Prometheus grabbing data from Kubernetes. Whatever way you have observability for kubernetes, the sidecars will be legible to it. But most importantly, it simplifies these container, right? Like if we want to go with the model of a container, it's supposed to be very simple. This is a way to offload the complexity to Kubernetes. These you want it and to have each of these containers, both the main container and the side container, each be much simpler than if they would somehow have to be carefully finagled to been the same container. So obviously the reason I talked so much about readiness earlier is that that's going to be one example for why you would need a sidecar. So for example, with a readiness sidecar, you can check that the file exists or that the file is pretty new. You can check the contents of an HTTP response. This is not something that's very easy to do with the Kubernetes management, but you can easily send an HTTP code, and now you write your own code. So you can write it in a nicer language like Python. You can have an arbitrary logic and you don't have to start shoving in shell commands into Yaml and have them execute in these kubernetes container and make sure that they properly offer and stuff. And there's a whole thing. This is easier, right? You can just write code in Python. Python is a nice language. You can easily check the containers, do all kinds of complicated logic. It's easy. The other nice thing that you can do with side cars, collect metrics. What if the original container wrote some code and that wasn't formatted in the way that your metrics management, say Prometheus, which most of them kind of are compatible with, can read well. You can grab those metrics via whatever reformat them into promotes. You can also do synthetic checks. You can send a query to an API endpoint, measure how much time that took, and that can be like a nice gauge or a histogram, depending on exactly how you do that. You can even scan a log file, check, say how many black lines are there. And that can be your metric, right? Like how many lines were written in the last minute. That's a useful metric that you can expose. And so there's a bunch of stuff that you can do that's like that, that you can just do in a sidecar. You can also do scheduled tasks, right? Like I already gave these example of garbage collection, or you can do some sort of a re indexing operation. Again, like the sidecars shares all the knowledge that the main container has. So it can directly go with the API and kind of grab the data, do whatever indexing you need on it, and then shove it back through the API. Why write side closing Python? So there's a lot of things that make Python an ideal choice for sidecar. So if you think about it out of your examples, if you really knew that you needed that functionality early in the development process, you would already kind of but it in the design and made sure it fit right. The problem is retrofitting it. That is hard, but that also means it's late in the development process. You don't have a lot of time left in your schedule. Luckily, Python is a great prototyping language. You can whip out something that kind of works very fast. So that rewards the ability of python to quickly prototype. Now, the main thing that the sidecars will do is to look good in some sense on the dashboard, right? That means that you want a very fast feedback loop. Again, that gives you a lot of reward for fast iteration speed, right? It's not just easy to write the first version of Python programs, it's also really fast to iterate. And Python has a lot of facilities that makes iteration a lot faster and also kind of the way politics in engineering works. Often the sidecar will be built by slightly different people than the people who build the main car, right? Like they might be know kind of the deployment team or the infrastructure team or the SRE team, people who kind of adapt to that, maybe an embedded SRE, which means that the container will keep moving and the sidecar needs to keep moving along with it. It's really nice if you have something where you can adapt as changes in the container cause whatever logic you had in the sidecar to go out of date to be faster. So this means that if you can chase the container code faster than the container code moves, you're in an advantage. If you take longer for your cycles to modify. When the container code moves, you're perpetually going to be more and more behind. So I think this is kind of the fun part of the talk where I will give you an examples. The challenge I gave to myself is to feed a sidecar on a slide. That does mean that the code is optimized for the slides, not best practices. This is not how I would actually write this nodes. In practice it's mostly to show what bits and pieces would make a sidecars work. And hopefully they're going to be fun because you can kind of get all the details in one slide. This also does mean the slides are going to be a bit packed, so get ready for a fun journey. So here's like a really quick readiness sidecar. As you can see, in order to raise an exception, I just divide one by zero because that's literally the fastest code that will these shortest code that will raise an exception in Python, just three characters. So things means that I can grab the data, compare it to, let's say five, and then return the response okay, if it's not five, I raise an error. Most web framework will automatically give you 500. That complies with kubernetes. I have just enough code to hook this function into pyramid. I have routing, that's it. Other than imports things is fully functional code that will give you a readiness check. Assuming that you have a JSon that has a key called value, that has to be five. So this is fully functional readiness sidecar. Here's a metric sidecar. It takes a little bit more code because I have to integrate with the Prometheus client library. So that means I have to build these collector registry and I have to build a gauge for latency. And then I basically send a synthetic request, check how long it took, set the latency and I generate the Prometheus thing. And again, if you set everything up in kubernetes correctly, set Prometheus up with the right configuration. This will give you a gauge that shows the latency of the application, which means if the application is heavily loaded or it's like garbage collecting or something, this will take longer and you will be able to see it on a dashboard and you don't even have to count on the application actually measuring itself at any point here. Here's a simple example of a scheduler side cars. All it takes care of just periodic making sure that the application flashing bits queue. You might think, well, why not just put a Kubernetes quant job? Well, a quant job is a job, which means it is a pod. But let's say that you want this. Every single container of the main application has its own kind of internal queue that you want to flush once in a while to make sure that it flushes this is fully functional code and it will actually do the job and you actually don't have a much easier way to do that. So three lines of python. This was very easy to fit on a slide. So those are like some fun sidecars. Now I'm going to fail in my challenge because the next sidecar I have is going to be slightly longer than a slide, but we will break it down to a few slides. So it's going to be my log analyzer sidecar. To challenge myself, I gave myself kind of like some fun parameters, right? Like I said, you can actually mount a volume and so share files natively. But let's say for any kind of reason you don't want to do that, right. It just doesn't fit your model. So there can be all kind of permission issues and look, weird corner cases where it wouldn't make sense. So the main container logs to valog something. That valog something is inside the container's file system, so it wouldn't be accessible naively to other containers. And you wind these sidecar that processes has a log in this example, I think it's going to count the lines, but obviously in real life you'd do something slightly more interesting. But this almost seems like an impossible challenge. So I think the fact that I can't fit it in one slide is maybe, okay, I will fit it in like a handful of slides. So the first trick is that to configure the pod correctly. So we need to share the process namespace. Process namespace. That doesn't sound very useful. So the first step is to find out why do you have to know yourself? Because you can't know others unless you know yourself. This sounds like a weird, like mystical philosophy, but I promise you this is actually concrete technical advice, because now we can look for another process which has a different root than ours. Now we managed to get to the other process roots, right? Which means we already have access to the peer Containers operating system. So that's pretty cool. So now we have access to the file system that's in the other container. Now we can just read the log, sum the lines, and then post it to some kind of push metrics collector, let's say. And that's it, we're done. So it was a bit more than a slide, but in three slides we did something that most people would tell you is impossible. So I think that is a pretty cool thing about showing both the strengths of the kind of stuff you can do with sidecars and the kind of power that you have with python. That you can do the impossible in three slides, not even full slides worth of code. So I think that's kind of cool. So I want to leave you with a few final thoughts about containers. They're black nodes, right? You don't want to mush around with a container. Like once the container has been tested, you want to keep it as is throughout the deployment cycle. Then you want to keep your container simple. The simpler the container is, the easier it is going to be to manage. But you will have to always add something. Pods and equivalent things. On other frameworks that run containers, what they do is they allow adding stuff from the outside. At least it's a clear ish separation of responsibilities. So that's pretty useful. You can build the container to be more sidecar friendly. That's actually more important than building a container, to not need a sidecar. And one trick is to just have good APIs inside these container. You can, for example, listen on a separate port that's not exposed outside, so it's only going to be exposed to your peers, which means you can do a lot less permission checking, but still define a really well defined API. Which means you don't care about security, but you do care about portability, so that it's easier for the sidecar container to know which API to expect. So packaging stuff with containers is pretty good. This is not a controversial statement. In 2022, everybody agrees packaging stuff in separate containers is even better. You don't want to shove a container that is like the everything container or you're back to like, why even have a container? So the more sidecars you have, the better your packaging architecture is. And with that, thank you so much for listening to my talk. I hope you had fun, and I hope you're going to go and write a lot more sidecars.