Conf42 Cloud Native 2024 - Online

Securing CI/CD runners through eBPF agent

Abstract

In this presentation, we will share some of our observations through showing different flavors of attack on possible development pipelines and introduce a tool to detect them.

Summary

  • Today we will be talking about CI CD tool that we developed to secure runners using Berkeley packet filters. There are a lot of malicious application in the wild which dramatically increased throughout the years. Common attack vectors for attackers to achieve this are vulnerable packages and compromised pipeline tools.
  • Nordic career based threat group recently published NPM package called execution timer sync. Package aims to steal the victims login credentials and passwords from the browsers. Also has a python script capability to further the compromise. We decided to develop our own tooling, which is also open source called control.
  • EBPF technology is a way to write kernel level applications without literally writing or compiling anything in the kernel. The tool will basically run any newer Linux kernels which most of the Ci CD runners will apply. The aim is to develop a full fledged open source ci CD security solutions with EBPF power.
  • So that was it for today. Please join or contribute to this project. If you ever have any questions, feel free to reach us. We are happy to answer or even contribute to your questions.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hey everyone, welcome to our presentation. Today we will be talking about CI CD tool that we developed to secure runners using Berkeley packet filters. And I'm Mert and I'm joined by my friend Jenk, who's the CEO and co founder of Kondukto. For introducing our tool, we will first talk about the trend landscape and to showcase why we're developing this tool. And then we will talk about why we choose Burkett Packet filter and how we developed our tooling approach, and then we will showcase a demo of the tooling. So if we look at the trend landscape today, as you can see, there are a lot of malicious application in the wild which dramatically increased throughout the years. And if you just look at the two years ago and compare it with the last year, you can see there is a lot of interest from attackers in terms of malicious packages. And as you can see, there are common attack vectors for attackers to achieve this, the first one being the vulnerable packages. They publish a vulnerable package using typoscutting attack or publishing a package that resembles a legitimate package name that is widely used so that they are trying to trick the developers to use those vulnerable packages instead. And similarly, they also took an approach of compromising pipeline tools so that they can tinker with them or they can publish again typoscarted pipeline tool, which might seem benign but might not be. And lastly, they can write out compromise the artifact integrity. So this could be a code or your container image, so they can target those and compromise them in order to for example, publish or deploy code of their own choice for crypto mining purposes or some other purpose to leave backdoor. So based on these common attack vectors, we can talk about, for example, one of the recent threats that is happening in the wild nordic career based threat group recently published, for example, NPM package called execution timer sync, which is a copy of a packet called execution time which is being used as a code profile profile provider. And this package aims to steal the victims login credentials and passwords from the browsers. But as you can see on the left side, it also has a python script capability to further the compromise. Once you have this NPM package, it collects the credentials and passwords, and afterwards it tries to download this python script so that it can remotely control the compromise instance in order to achieve this. On the right side, you can see the diff of the recent build that was discovered by Filio. They're constantly changing the IPs to stay hidden. And this has been. If you follow the blog post, you can see that this has been tied to North Korea. And this is not the first time that some threat actor group is trying to masquerade as a legitimate package. And that's why when we see similar threats in the wild, which can affect also ci CD pipelines in form of malicious packages or compromised pipeline, we decided to develop our own, our tooling, which is also open source called control, and now jenk will deep dive into the package filter. What is it and why are we using it? As well as some of the use cases that we looked at to implement within our tool to monitor and prevent these kind of attacks. Yeah. Thank you mark. So welcome everyone. So in the second call section of this talk, I will explain why we have chosen EBPF technology to implement this open source security solution. So EBPF technology is gaining more and more popularity in the observability and the security space. And we decided to use EBPF because we want to monitor and prevent attacks or anomalies in the runtime. And since BBPF allows us to write portable applications because of the compile ones and run everywhere approach, this was obvious. And let me explain you what is EBPF? Let me rephrase once again. So EBPF is a way to write kernel level applications without literally writing or compiling anything in the kernel. And because of this portability approach, when you develop such an application, this tool will basically run any newer Linux kernels which most of the Ci CD runners will apply. Ebpf basically lets us develop a user space application while modifying and touching to the kernel space. And as you can see in this example, literally writing in small hooks in a specific kernel section or kernel syscall, we can either asynchronously see or monitor the props or even modify for some of the. So we asked ourselves, so, how to develop and what to do while developing those securing solution. And we explored the options. And when we have seen all the attacks, and when we have analyzed all the attacks, we have seen a common pattern which was the network layer. Obviously we can develop more and more controls and security tooling using this technology. We started to develop this open source tool with the traffic control layer. And how the system works or system has been designed is whenever a new event occurs in the pipeline, we are going to attach appropriate kernel calls. We create separate sections or channels, send those probes into the user space, and user space will act as a decision making or policy engine which will allow or prevent or block the relevant access code. So in the latest attack that we have seen, we have observed the north korean one, the Lazarus group. This design was pretty variable. And if the vulnerable package has been triggered in the developer's machine, it was acting differently than the on this other side it was running in the Linux section there was Android approach. So that's why we have developed the monitoring capabilities as well as the privileging capabilities and added those controls in the tutorial. So what is the playbook? So playbook, as I explained, first we should decide which events or which books we are going to attach. Obviously we wanted to have the TCP and the UDP connections, so we are attaching the TCP V connect and the IP four datagram connect calls. Those calls are creating dedicated channels into our user space and our user space or acts as a policy engine. And whenever an malicious event has been identified, we were basically telling our filtering function or filtering to prevent the access. So this is the overall picture and our aim, our idea is to develop a full fledged open source ci CD security solutions with EBPF power. So let's jump to a quick demo. Then let's explain how the system works. Right now we have developed this as a single binary. So this means that since the single binary, and again because of the core architecture, it will work basically in any Linux kernel. So as you can see here, on the left side we have run the code, and on the right side we try to get some HTTP calls from the Google and then the Amazon. Since those are blocked, we see that those tcp communication has been blocked already. And we then created a wallet call and download the artifact. And as you can see, those TCP we get were successful using this approach. Let's show us how the same approach will work in the GitHub runner on a GitHub action. As you can see here, we have developed a pretty simple and pretty valid GitHub action file. So we have two actions here, or two steps. One step will act as a malicious event, and the malicious event or malicious step will simulate and secretly cooperation. And on the other end we will have a load action which will download the artifact or which will upload the artifact. And finally we will see the report, we will just see what's going on. So we are triggering the event, and since it's a single binary, and we have developed and dedicated Docker as well, so it's dockerized. You can run in the docker format as well, based on the CI CD runner. So control binary has been developed and one action prevent action occurred. As you can see, the network unreachable, and in the allowed action it works. And down below we have a report that works as expected. What was the idea behind this tool? Obviously, if we can monitor the network or what's going on in the pipeline itself, we can take certain measures. And we decided to develop this on an open source because with the help of the community, security community and EBPF community, we believe we can implement more and more sensors and the latest version or the new version of the tool supports UDP protocol as well, and based on the configuration it can prevent access or allow access to the GitHub meta IP range and in progress feature we have developed an open policy agent based rule engine. It's been tested and we will publish soon. And afterwards we will create an prop monitor process monitor sensor to understand which files has been accessed or even touched, and we will get a list and we will try to find anomalies in the file system using the proc sensors. So that was it for today. Please join or contribute to this project and if you ever have any questions, feel free to reach us. We are happy to answer or even contribute to your questions. Thank you. Thank you for having the call.
...

Mert Coskuner

Principal Product Security Engineer @ Yahoo

Mert Coskuner's LinkedIn account

Cenk Kalpakoglu

CEO & Co-Founder @ Kondukto

Cenk Kalpakoglu's LinkedIn account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways