Chris, I've been used, trying, at least with some realms of success, to use Kubernetes. Since zero four. I've got opinions, so strap in. I'm a solution architect at Via, instructor at learn k eight and a tinkerer of a bunch of open source things. I often talk too fast, so key kick me or something, or throw things at me, it's fine. We're actually human people that we can do that now in a minute. So with a bit of audience participation by show of hands, who's worked with Kubernetes pods before? Anyone? Hey, in the right room. Good. Look at that. Actual hands and people. Cool. And for anyone who didn't raise their hand, well, welcome to party. You've almost missed it. Pods are the smallest deployable units of computing that you can create and manage in kubernetes, and they represent like a single instance of an application running in your cluster. Okay, so that's the topic of this. Well, what about pod security policies? So they've been around since 1.0, which is about a million kubernetes years, and in that time they've never made it past the beta classification, which is a curious observation of its own. Right. And I believe it may actually be the last v one beta, one resource that's often routinely used in production since ingress was recently matured to a v one. Recently, however, sadly, that's not going to be the case for psps. They were deprecated in 1.21, which was April last year, and will be removed entirely in 1.25, which will be around August this year. So what is a PSP, apart from more words than should ever be on a slide? That's better? Well, PSPs give cluster admins basically, in short, an ability to impose some limits on the things that are running in the cluster. So things like being able to run as root, opening ports on the host, types of volume you can use, et cetera, if you've not seen one before, it looks something like this. So by show of hands, or actually put your hand up and let you volunteer an answer, but who could give me an example of what I could do with a container that looks like this? The pod spec. You can see that without any reflection? All I can see is glare. Go then. Not much. Not much? It's an altime image. It doesn't have anything inside. Not much, you reckon? No. You can run it as root, but you don't have tooling inside. Okay, any other answers? Salmon, stop feeding our answers. Okay, so I'm not going to bother I won't do it with Alpine, but the thing that will actually the example I've got is pretty much the same and I don't use any of the tools, so I could have done it with Alpine. So let's flip the other way. So I've got a hopefully. What's that? Zoom in, needy. Is that better? I mean, you won't be able to read anything now, but sure, I've zoomed in. Okay, so I've got a cluster, right? So I've done it with kind for the sake of doing this easily. But you should be able to get the idea right. It's a cluster that's running, that doesn't have any pods. It's vanilla as it comes. Right. So I'm going to do something reasonably straightforward. So I'm going to do it. The Debian image says, your point doesn't make any bloody difference. I could do exactly the same things with Alpine, but I'm going to do it with Debian just for the sake of it because that's what I had prepped. But it is a vanilla debian image. I'm not using any of the tooling that comes in the image. Now this would be the same whether it was Debian or Alpine. Does that give any clues to what I could do? And I've demoed this to a load of app here, folks. So you're not allowed to answer any clues. So what if I were to make a directory? So make like slash, foo and then mount one of those volumes. So you'll notice that the amount of things I have here, that's the host. Right. So this is a VM that's running on my laptop. If I can not do that, sorry, I missed the last bit off the thing. So I can mount that now if I look in, well, I'm now up to the host. Okay, so what can I do with that? Well, I can do some interesting things. So without tooling, as you pointed out, I can export path and just load into my path, find where tubecuttle is because I'm lazy. I could use curl or I could port to the devtcp port and tunnel it through. But I'm lazy. I'm just going to use the vanilla tools that we're all used to using like kubectl and things like that. So I do that. I'm going to just sim link in. So I don't need to mess around with pointing in the right places. So I'm going to sim link in the kubelet. So I'm now on the host. So I can just use the Kubelets certificates to talk to the cluster app as the cluster API, and I'm going to set my kubeconfig to be the Kubelets configuration. So now what can I do exactly? Right, I've poned your cluster. I can go and get all the pods. I can go and get all the nodes. I can pretty much do whatever I like. I have pretty much most of your cluster now. I can't exec into a pod easily. What I can do if I do say kubectl get pods. So this is the one that I'm already in. But if I try and do Kubectl exec and then try and run bashing it, right, I'm going to get a no, because the Kubectl API says that the kubelet, quite rightly so, shouldn't be able to exec into your pods. Well, sure, but I can do a load of other things. So I can go and relabel your node, so I can pull all of the current workload to the node that I own, so I can talk to the docker demon or whatever containers runtime you're running and interface directly with that, right? So the docker socket is all available to me. So I can go and look at whatever the containers are running on the node that I've managed to get, and I can go and drain all of the other nodes. So I can do like a cubecuttle or cordon nodes. I could go and drain all of the other nodes, and I can label myself with all of the taints and tolerations that you might have had existing. Because I've got the ability to declare who I am in the cluster as a node, because I'm impersonating a node at this point. So I own your cluster. I own everything that's running on it. I own everything that's outside of it that it can connect to. Sad times, right? Yeah. Don't let me have admin on your clusters. I guess it's the message of the story. Okay, a bit more obvious, but again, by doing something like this, game over, but pretty much under the same terms that we've just covered. How about this? Can anyone help me out? What might I be able to do with that's okay. So I can masquerade as other things to the network. So I could maybe open up on port four, four, three or 80, depending what you're running on your network, and potentially impersonate what might be like a normal ingress service. So if I can boot out your ingress pod that's locking that port, or I find a way to crash that, I can maybe hog that port on the host port again. I can now sit between traffic between the outside world and your apps. I can also talk to and masquerade with all sorts of local systems to the host, like x or D bus ice guzzy, depending on what your world is, or indeed the container runtime as well, depending how that's configured. Bit more involved, but ultimately game over after you've basically just jumped through a few more hoops in order to actually get that route. So basically, leave any one of those controls in the PSP open and you can be kind of one bad pod away from losing absolutely everything that you own. So psps fundamentally sound like a great idea, right? Well, not so fast, bucko. There's a whole heap of usability issues that you might have encountered if you've actually tried to use them in anger. So the policy is based on the user that created the pod, not the workload. And if you've created pods with, say, CI or something, for example, that might require you to have multiple identities to authenticate to the API server with. But fundamentally, when was the last time that any of us actually created a pod and not a deployment or a stateful set or a batch job or whatever else, right? So, well, in that case, the identity creating the pod is the service account of that controller. Some of the parameters aren't simply admission controllers that accept or reject, but actually do some manipulator, some mutation on the actual pod itself. And that's not clear from the docs. And the order of the evaluation also can be confusing and in some ways unpredictable, with multiple policies with overlapping scope, especially if there's some of those mutating ones. And it only applies to new pods, not anything that's already running on your cluster, which means you might not know when you update the policy, and then it goes and breaks something that you're already running, and you don't know that it's going to break it until it happens to get to a point of rescheduling. So maybe a node fails or something like that, say that the pod is going to reschedule. So, well, what are the alternatives, right, in this new world? So what should we do? Well, the clock's ticking. August is probably only another lockdown or so away, so there's a fair amount of choice. And here's just a few. I made them shine and like all the rest of it, right? So they're shiny. Or you could write your own like it's just a webhook fundamentally that you can provide a 200 or a not 200 response to. I'm going to focus on a few because with a little help they provide like a straightforward ish migration journey. There is an intrigue answer, by which I mean it's built into kubernetes, so why am I not just kind of pointing at that, grabbing my drink and walking off. Right, so pod security standards are most easily thought of as three rigidly defined predefined pod security policies. So those are privileged, which is basically anything goes. So it's the same as like not defining a policy at all baseline, which is like a middle ground, stops some of the super obvious stuff. Most your stuff should probably, I'd imagine, probably run at this stage, at this tier without change and restricted, which is the most restricted policy and stops most things. So I would generally suggest aspire to run stuff here, but realistically you'll probably have some issues. So rigid universal policies sound great, and it's super easy to communicate these between teams, test against them, and absolutely no confusion when you're deploying between different clusters. So, happy days, right? Well, sadly not for a few reasons. So ideally you'd run all your workload restricted, but realistically inevitably there'll be some things that can't quite fit. Well, restrictions are applied on a namespace level and there's no way to grant fine grade exemptions. So your only option then is to take a massive step down and take that entire namespace down to the next kind of significant step beneath that, and it's applied with a label on the namespace. I mean not even like an annotation or an extension of the namespace policy. Fuck's sake. So what's it good for? Well, the only thing I can see this is possibly good for is if you're a software vendor building products that run on Kubernetes, and if you're mad enough to be in that business, then if you can make your product run in a restricted, then it will probably give you a good head start for whatever unique unicorn features that your customers will have implemented and demonstrate that you've at least considered the security implications of your product. Okay, so how do we migrate all those old legacy psps to something new? Well, it's unfortunately no simple like for like mapping pod security policy. And any of the replacements behave slightly differently and fundamentally for good reason too. So if you've been using pod security policy for a while, you probably likely have some quite complex rules in place that have become entangled with some of those usability issues that you might have. So I would generally just encourage you to take the opportunity to then refactor those and simplify trying to continue, rather than trying to continue what you've always done up until now. That said, there has been some work to ease the transition by reproducing the same capabilities and even calling them the same things in some cases. So the short answer is, use our whizbang super duper tool. I should have worn my pink jeans for today, sorry. So with this, you can simply provide your existing PSP and take your pick of policy engine from Conervo keyboard and or gatekeeper, or just paste it into our simple web app and let your browser do the work. So to demo that real quick, we can do if I can remember the URL to it via GitHub IO sorry, it live demos. Let me get to the address ow with that. So with this you'll notice the big report Abug button, probably taking more focus than anything else. And that's important because yes, making bug reports is probably good because it's probably not perfect. It's definitely not perfect. I built it with this. I'm just going to pick like an example. So you'll see the policy on the left hand side, and you can see on the right hand side the policy of what that looks like in gatekeeper. So if you're using OPA or in cubeboarden, which is a bit bigger, or converto, which again, errors. So yeah, bugs are good, right? If I do that, it might do something different. Okay, fine, bugs, cool. Raising bugs is probably a good thing. If I do that, that works in canoe at least. So yeah, some of them don't work, but that's indicative of some of the mapping and some of the limitations in those policy engines. Right, okay, fine, here ends the live demos. So by doing that, well, your PSP just converted more or less, as we saw seamlessly to canervo keyboard or gatekeeper was easy, right? Well, but should you. So don't be fooled. Just because I wrote a tool to help does not mean I think any of this is remotely even a good idea for the vast majority of the use cases. But it got your attention, which is the important bit. Using a cluster enforced policy does not guarantee any real security. You may well find that the cure is actually worse than the disease. So as a trend, we're seeing more and more vulnerabilities in the wild with attacks that cluster enforced policy would actually not protect against, say, this and porn kit and dirty pipe and dirty cow and similar silly brand names that cves get, but they're more in the case. So if you've been keeping up, you'll probably realize I've taken you on a roller coaster. So I've explained a problem with security and kubernetes, a solution that was PSP, another problem of psps going away, and then another solution of a tool that I built, and I'll get another problem that undermines everything that I've just told you. Well, the good news is there are answers. They are simple, but not necessarily easy. But I'm out of time, so you'll have to come back and find out from that in my next talk. However, as a sign of some good faith, you can expect a scenic walkthrough, some buzzwords like these, and I'm excited to be able to share some of the hard but fundamentally simple solutions that can actually provide a robust level of coverage, and also advice on how to tackle the cultural changes that need to go hand in hand with the tech. Thanks for your time. Hopefully this has been interesting, if not just a tease. Please do follow me on LinkedIn, Twitter, all the other nonsense at GitHub. You can be assured there'll be no spam, since I'm simply awful at any sort of self promotion, especially on social media CNs me just points at my LinkedIn at Apvia we're doing a ton of open source, so do check out both mine and Appvia's GitHub orgs star and all the other things to watch and to your heart's content. The original content for this talk and some of the solutions I've alluded to, including how to do policy as version code, are available on the Appia blog at the minute questions, as Graham said, we can tackle later. Otherwise I would have held the stage, which was the notes that I had. And that's me.