Conf42 Cloud Native 2023 - Online

What We've Learned from Scanning 10K+ Kubernetes Clusters

Video size:

Abstract

Kubernetes security is still hard, and it is possible to get a better understanding of your security and risk posture, and take measures to improve it with open source tooling.

Summary

  • 96 of the organizations are using Kubernetes and we see it's growing over the years. The person who is responsible for security when it comes to secure your cluster is the DevOps. The second one is the runtime protection. Kubescape is an open source solution.
  • 63% of clusters have workloads that are exposed outside the cluster without any ingress block. 23% of the cluster had like application running with dangerous Linux capabilities. 46 of container had one or more critical vulnerabilities specifically. If you have any critical vulnerability is mandatory to fix.
  • Most of companies already run like Kubernetes clusters and security is a big pain, but no one really wants to deal or handle it. And as I said, DevOps are the new security personas. And we are overwhelmed with misconfiguration and vulnerabilities.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hi everyone, welcome. My talk. Right, so before I'm drilling down to statistics, I want to show you what does it mean to have security in kubernetes at all? Because what weve end is in the matter of security perspective and not just in general, of course. So kubernetes is now de facto what is called the new cloud native operating system. And that makes it a lot of companies. We see that 96 of the organizations are using Kubernetes and we see it's growing over the years and like in this year, mainly a lot of adoption of kubernetes. Mainly this is because it's, I would say, the only orchestrator available today that is managed and everyone is fully aware of. So having said that, the Kubernetes is the operating system. What do we need to do? We need to secure our operating system because attackers don't no longer see it off the radar. They do try to bridge in the Kubernetes. We haven't seen any major bridge, but we do see a lot of tries and we see a lot of documentation on that. But still it hasn't happened yet. Hopefully it won't happen at all. But we need to secure SF in that case. So what we also see in that matter is that everyone is approaching to the security Persona in the organization, to the CSO or security manager. But what we do see is that he is not responsible anymore for security when it comes to Kubernetes. So we do see that the person who is responsible for security when it comes to secure your cluster is the DevOps. Right. We can see it right here. The. Right. So the DevOps is responsible for the security. DevOps, devsecops, any Ops Persona is responsible. And we do see it comes in often cycles like releasing weekly releases. And they are using Kubernetes specifically to improve the scalability. We're using restraiser because we need to improve the scalability and also we want to decrease the development time. So those are the major point of why even using Kubernetes and who is basically the Kubernetes security role in your organization or in any organization. So Gartner checked that and said, why do we even need any Kubernetes security at all? Right, because I'm often hearing that, okay, oh, I'm on kubernetes then all good, I'm secure. Great, that's wrong. We do see that DevOps are telling us that, or telling Gartner that the security incidents and issues into their kubernetes cluster are mainly into detected misconfiguration like they misconfigured or the YAML are default values sort of things. And on the other hand, what they're worried most is cves in images like vulnerabilities and again misconfiguration that can expose to let's say outer network and so on and so forth. So having said that, we have like two paradigms of how to protect Kubernetes. So the first one is what weve calling kubernetes, posture management. The second one is the runtime protection. So the posture management is to find known vulnerabilities and mixed configuration. It's something that we know in advance, but the runtime protection is like anomaly behavioral analysis and network segmentation. We don't know it ahead of time. So what do we focus in on is in one hand like tricking the attack surface, and we can detect the posture management early in the CI CD pipeline. And when coming to runtime protection then we want to assure and actually take an action upon detecting in runtime if attack has happened. So in order to drill to the numbers and statistics that we calculate and got, then we needed Kubescape, which is armo's open source for Kubernetes security. The architecture is pretty simple. Kubescape as a whole is an open source solution. It gets from the cluster, the vulnerabilities and the configuration issues also from the git repository and container registries. But we won't focus on that, that time. And eventually it sends all the data to arm platform. Arm platform is where I can see all this data in a very nice way, I mean UI visualizing way, but also that's where I'm storing the data and I can analyze that. So I will talk a bit about Kubescape because that's the tool that got us all this information and what we've learned from scanning with Kubescape. So Kubescape really became like the majority of tools, most popular tools in security, open source security tool for Kubernetes. And it got really high lately. And what we see here is, remember saying that we need the posture management and we see the runtime management. So here is part of that. We can see the risk analysis and compliance, we can see the RBAC visualizer, like what are the roles and responsibility, a visualized graph of your RBAC in the cluster, and of course the vulnerabilities coming along the way in your images. So we're talking mainly about the CI CD pipeline. So we need to check everything at the early stage of the CI CD because we don't want to get into the cluster and realize that we are having some misconfigurations, right? So let's check things ahead of time in your repositories, in your registries, across the pipe, before getting into the cluster. But when weve in the cluster, we need to verify that we don't have any misconfigurations there, of course. But we would like to take the other step of runtime zeros trust. So it's super easy to install, it's just a help chart. Just run the command and you can see it all in the GitHub, very easy. And of course CNCF sandboxing recently, so you can see it right away. So after having this understanding of what we gathered and how we gathered this data, so now what we're learning from that. So, as said before, what is the user title of those people in the cluster? Like, I think around 60%, even more like 60, let's say a bit more, are DevOps. So DevOps is the security Persona that we defined in the data we collected from the clusters. And the region is like around North America is half, Europe is around third, and all the rest. And let's see another overview of what data do we have. The number of clusters. Most of the users has one clusters, 33% has like between two and four. And the cluster size, I mean, how many nodes are in this clusters? 34% are less than five and 33% are between six and ten. There are of course some beyond 50, but of course it really depends on the environment they are scanning. So what have we learned in Kubescape? We are scanning by framework. We're scanning the NSA framework, the Mitre framework, the armor best, and the DevOps best framework. And the average score per framework is around 30%. And the score that is lower than 30 is an okay score. Okay, beware. You can check your own score with kubescape and see in which compliant framework is the score available. So the top five misconfigurations are run privileged container, cluster admin bindings, missing resource policies, immutable container, fast system, and ingress and egress blocked. So what we do see here is 100% had at least one misconfiguration and 65% had at least one high severity misconfiguration. And 50 of the cluster has more than 14 failed controls. Controls are the substitute of the framework I talked before, and those are the misconfigurations that we are testing and finding. So let's drill down a bit to what we see. So, running privilege container is a big no no. And we can see the basic of what users are running. And if you are putting some data into the pot spec, then we see it's not privileged, right? We can see that in the security context if you run its user that is greater than nine nine nine. And the runners group also will be greater than nine nine nine, you're pretty much safe. But you also need to have the allow privilege escalation as false. So privilege container will run in your pod. Okay, that's big bridge into your cluster. You can check that out also in a blog that Banda C armo just wrote. So the next thing is we talked about 63% of clusters that have some workloads that are exposed outside the cluster without any ingress block. So of course we do need to check those data, you need to check the ingress. And in Kubescape we do check that. But look how many cluster and look at this percentage. That means that the tunnel out is clear. And also the 63 again percentage of the cluster had workloads without proper resource limitation. So if you don't have resource limitation then each container can take as much resources as he wants, right? He can take the entire clusters resources if he needs. And that is not just a developing problem but also it's a security breach. And 37 of the cluster has like application with credential and configuration file. So it's so easy to say that. But we do see that going and repeating all over. We're checking for password word, we are checking for JW two, we are checking for private key. All those kinds of phrases are still available in the configuration files and 23% of the cluster had like application running with dangerous Linux capabilities. And that's like basic because also if weve running on vms then we need to harden our application. So again it's not just related to Kubernetes, it's just manual thinking. Also previously when talking about passwords and all those other phrases then it's super easy to check and also to solve. So 35% of the cluster had workloads running with insecure capabilities. So we're checking both of those things and together we see like it's majority of clusters. So if we will take a look now onto the vulnerability part of the cluster and we checked for vulnerabilities. It's a good thing to say that has zero critical vulnerabilities but they do have one or 217 percent critical vulnerabilities, et cetera. Again that's a breach. So if you have any critical vulnerability is mandatory to fix, especially if it's a remote code execution one, it's super important one. And the vulnerability by severity again is just another view of the critical and high end, medium and low. So the top five vulnerabilities we found I've seen here and what is interesting, that 63% of the container had one or more vulnerabilities across and 46 of container had one or more critical vulnerabilities specifically. And we talked about RCE remote code execution that we see 53% that container had like one or more RCE vulnerabilities. All that we learn from scanning just this ten K clusters. This is the information about the critical vulnerability. Glipc of course, and Lib Cre. There are very well known cves, you can read about it also in this blog. It's super detailed about the container scapula and its impact on Kubernetes. I must say that we will get into it in next talk. But when I'm thinking about what we really learn and what are the conclusions that I'm taking from that talk is that most of companies already run like Kubernetes clusters and security is a big pain, but no one really wants to deal or handle it. And if you're taking it to the security Persona, he says no go to the DevOps. And if you're taking it to the DevOps then he said no go to the security Persona. That's something that we should combine together and therefore we see a lot of devsec ops operation now. And as I said, DevOps are the new security personas. So Devsecops is the new phrase. And we are overwhelmed with misconfiguration and vulnerabilities. We do see it's a big problem. And here in Armand we are trying to figure that out with relevant vulnerabilities specific for the cluster. Whatever runs in runtime and not just detected in previous sets, whatever is really, you know, onto the, onto the memory at a time. And this will be continued and we will show it next talk. So thank you very much for being with me here today. It was a pleasure and thank you.
...

Rotem Refael

Director of Engineering @ ARMO

Rotem Refael's LinkedIn account Rotem Refael's twitter account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways