Conf42 DevSecOps 2022 - Online

How to implement a DevSecOps pipeline with zero-trust

Video size:


The purpose of this talk is to discuss how to implement a full scale Developer Security Operations into a DevOps pipeline secured with a zero-trust network.


  • Evan Gertis: How you do anything is how you do everything. Proper leadership comes down to planning, accountability and commitment. There is a high demand for infosec leaders who can pass the torch. Here's how to implement a developer security operations program.
  • A process and a team make a better dream. The weakest element in any process is the human. Passion, accountability, clear communication, execution. These are the four things that make an effective team.
  • We implemented full scale cloud security coverage for a mid sized organization. We used a free compliance framework in terraform infrastructure as code. This exercise is perfect for a new security engineer. We want to influence team accountability, and using this process, we can show progress.
  • Our process leverages x ray vulnerability reports in a simple way that can be used by any organization. The specific way that we use x ray reports influences accountability. Real leaders show they don't tell. Using this process we can show progress.
  • With consistent scanning, what we noticed was the vulnerabilities in the application went down because the results were so overwhelming. Do you want to save time and money by implementing this program? For more information, please reach out to me.
  • Evan Gertis is a Devsecops engineer. He graduated from the University of North Carolina at Chapel Hill with a bachelor's in science and physics. He also has a master's in computer science from Georgia Southern University. He is a certified security plus information security professional.


This transcript was autogenerated. To make changes, submit a PR.
Our name. Hello and welcome to this talk on how to implement a developer security operations program. I'm Evan Gertis. I'd love to start this talk out by just asking everyone, how do you feel today? I hope you're feeling well. Here are the five things that I want you to walk away from this talk with. Number one, automated scanning removes the need for manual labor. Number two, when application security reviews are executed properly, they increase accountability, but they require a security champion. Number three, strong vulnerability reports are more easily accepted when they are supported with easy to read metrics from baracode dashboards and JFrog XR reports. Number four, penetration tests fill in the gaps where automated scanning fails to catch vulnerabilities. Number five, the results of the penetration tests need to be supported with systematic logging. So why did the IT team set up their remote office on the beach? Seems cloudy. Ask yourself, why are you here? Prepare your mind for an extreme paradigm shift and let go of any preconceived notions that you previously held. What I plan to propose will destroy any of the accepted ideas that you once thought to be true. Imagine, what if we could give people a pill to get them to do their jobs and execute successfully? How you do anything is how you do everything. Proper leadership comes down to planning, accountability and commitment. This is what worked for me. Maybe you have different tooling, but the concept is still the same. You need to find the right tools to help you succeed. You do not need to do it this way. By following this process, you will be able to convey your results such that they will be received like a seasoned salt hitting your manager's tongue. I'm going to present to you what worked for me. We inspire your teams to take accountability. Our process uses the right tools to get people to take action. Our process is based on research from the two most referenced papers on devsecops from connected papers. We think we know culture, but we live in a world where the norms diet information in silos. And as Peter F. Drucker once said, culture a strategy for breakfast. How many of you have sat in a meeting with a liar? How many of you sat in a meeting with a micromanager? They can barely do their jobs, and yet they're going to tell you how to do yours. There's a reason they say crimes often come back to their teachers. It's easy to catch a lion tiger by their toes. These types can barely keep their stories straight from one day to the next. They don't have the common sense necessary for leading a team. This process does not make your employees into leaders, it inspires them to lead. We all know that most people can't do their jobs and as Benjamin Franklin so eloquently put it, always taking out of the meal tub, but never putting back into it. And what's worse is that it's both a technology and a people issue. So, have you heard of shift left? If you have, are you doing it the right way? Right now, there is a high demand for infosec leaders who can pass the torch. And Devsecops is a new field. The ground has barely been broken in. There aren't any standards in place that present a synergistic combination of people in process. So we can go ahead and pause here. Right configuration management it's been one of the biggest headaches for security engineers. How many of you have stared at a vim config file for 3 hours? Here's how we can address these issues. Provide process, implement structure, and finally lead our team. So, do you want to win win? Then clap if you want to win, win. Please stand. If you want to win, win. And if you're a real firewalker, go ahead and pound your chest. We make firewalkers. Take a second. Ask yourself, when was the last time you felt like your boss showed humility? Other processes do not care about your feelings. We convert your security teams into firewalkers to give your business an edge so that you can land and expand, I. E. We take you from being interested to a mature devsecops capability. Our process cares about your feelings. At the end of the talk, I'd love to give you a more detailed overview of the training presented here. I want you all to walk away from this talk with an understanding of the five key points that we covered at the beginning of this presentation. And if you want help implementing this, this is where I come in. As you can see, team members need to be involved in the early stages of the process. We need the right tools and the right people working on the right things at the right time, supported by the right reporting. The details of these reports matter more than the reports themselves, and they need to show where we are going, how we're going to get there, what needs to be done, and how we will know when we have arrived. The distinguishing characteristic between our work and the two most referenced papers that describe approaches towards implementing devsecops is that not only did we implement a pipeline, but our process leverages the human element. Most organizations miss out on this aspect. The weakest element in any process is the human, and we provide the appropriate structure to get individuals to take accountability for their work. As you can see, you need a champion. These things are more than just a technology issue. They're a people and technology issue. Why are these papers important? Because they emphasize that we need a bridge. We need more than just technology. We need firewalkers. It's simple. A process and a team make a better dream. How would you like to be 80% more efficient? A process gets a big dog's attention by bubbling problems up to the top without having to say anything. I walk through the fire and the flame. I dealt with difficult individuals, and I humbly set aside my agenda to hear what they had to say. And the key to influencing others is to listen. In order for our teams to be successful, we need to be able to establish common ground. We have to get team members to work with each other so that we can influence organizational change. Once we can show progress, instead of telling everyone what we are doing, it will become clear that things are moving forward. Too many managers fail to realize that the best way to influence a team is by showing, not telling, we have the technology. But the key is to build security champions so they can go on to lead. Others either get top shelf tooling or suffer the consequences of trying to cut corners. Let's face it, what's worth paying for is worth paying for. End of story. Our process leverages the best tooling. It's simple. Passion, accountability, clear communication, execution. These are the four things that make an effective team. We've got the Neosporin for your security needs. We're going to cover Veracode, burbsuite compliance, elasticsim, JFrog X ray, and a custom application security review. Here is what I'd like you to walk away from this section with. Our process uses a combined approach of static application security testing, SAS, dynamic application security testing, dask and container scanning. Then we use executive dashboards to support our claims. The combination of the tools and dashboards is one of the best ways to articulate our results to upper management. We take you from chaos to clarity. Here you can see an organization with over 254 Jackson databind vulnerabilities sitting in their repository. And then, just after six months of using this program, not only are those vulnerabilities gone, but all the other ones that you saw there are gone too, and it continues to decrease. First step adversity is the foundation of strength. And if you aren't first, you're last. You have to set the tone for your team. I did this myself, and I made it look easy because truly nobody wants to lead. We all know that success breeds success. And if you don't know, now you know, you have to define your champions. However, it takes more than just talk. The devil is in the details. The key to a good security report is being able to show the details of a security scan. Static application security testing will allow you to pinpoint the flaws that are plaguing your applications. The next step is to show developers the flaws so they can take the responsibility for fixing their vulnerabilities. And when you go into that meeting, you need to show the results and say nothing. Let the silence consume the room so they know you are serious. You want the results to speak for themselves. Simple. Go to the platform, select scans, analysis, and static analysis. Simply just using static analysis isn't enough. We have to use a combined approach of dynamic application security testing and static application security testing to convey meaningful results. Combined together, they actually produce something worthwhile. A simple overview of dynamic application security testing just shows that you create the dynamic scan, you configure it, you run the scan, you view the results, and then your developers fix the flaws. Now again, it's simple. Click scans analysis, dynamic scans, and launch the scan. The container is a foundational layer in any application. We have to bake security into applications by implementing container scanning. With bare code container scanning, your team can catch the third party vulnerabilities in your application. Then developers just need to click the fix button in the platform to see the detailed instructions on how to fix the issue. Here's one line of code. Throw it in a bash script. Run it. Done. The purpose behind implementing veracode dashboards is to show upper management the results of the security program. The dashboards will help executives understand where we are in the certification process, as well as the vulnerabilities that are plaguing our applications. So why are we doing any of this? It's to influence team accountability, and using this process we can show progress. Real leaders show they don't tell we're raising firewalkers. Burp suite is a sword for security engineers. It is a man in the middle attack platform, and we can use this tool to analyze and infiltrate web traffic. The only things you need to know are the burp proxy, burp scanner, burp repeater, burper intruder. We can use this custom pen testing workflow to find vulnerabilities that the SAS and dast miss. The Burp proxy is an effective way for one to perform reconnaissance on an application. This is one of the first steps in executing a great penetration test. The proxy allows us to capture requests being made to the application. But guess what? You got to learn how to crawl before you ball. The Burp suite scanner is an effective tool that will allow us to crawl an application so that we can define the scope of our penetration test. We can download a PDF report generated from the Burp suite scanner that will allow us to examine what is wrong with our application. It will provide us with the specific details that are necessary for conducting a proper penetration test and then simply just use these findings and give them topper management to support your claims. The burp repeater is a tool that can be used for intercepting requests and then modifying them with a custom payload so that we can attack. The key is using the intruder. The burp intruder can be used for attacking an application. It allows us to execute an automated attack against websites that we can rinse and repeat successful attacks against them. And if the attack doesn't work, then we can always iterate through another set of attacks again. And as you can see here, I am popping SQL injection and I'll do it again. And I'll do it again until the cows come home. Here's an example of a custom payload being built and running an attack. So why is this important? We want to influence team accountability, and using this process, we can show progress. Real leaders show they don't tell compliance. Most organizations don't even have anything in place. We implemented full scale cloud security coverage for a mid sized organization, and we used a free compliance framework in terraform infrastructure as code to achieve this. This exercise is perfect for a new security engineer. The center for Internet Security Foundation's Benchmark is a great compliance policy to start out with. It provides specific instructions on how to audit and secure systems, and it's helpful for entry security engineers who need to create a security program from scratch. The instructions are easy to read, and it's free. As one of my greatest mentors once said, you can build a river in a desert. So let me take you from this desert to filebeat pre configured dashboards. Here you can see all of the instances this organization is running. We have cloud trail as well, so we're monitoring the API. Oh yeah, that SQL injection I was popping, that won't happen on my watch because we have systematic logging to support the penetration tests. You can try to hide your bugs, hide your flaws, and hide your vulnerabilities, but we're secured everything up in here. The first step here is selecting that compliance framework. And as I said, you can't go wrong with the center for Security Operations Benchmark because it's free to start out with, why wouldn't you pick it? We can use the standard AWS resources to manage access control for an organization. The AWS terraform provider has specific terraform resources for managing infrastructure associated with AWS IM. Amazon S three is also a big part of this. It's a storage management service that can help us store data, such as Amazon load balancer ALB access logs, which can be queried either using Amazon Athena or leveraged as a part of the pre built Filebeat AWS module that I just showed you. Elasticstack can be used as a security information event management system, a SIM for monitoring an organization's cloud resources. To get started with this, we need to implement logging with cloud trail. Once we have that configured, we can move into elasticsim. We can take advantage of the AWS filebeat module along with the center for Internet Security Operations Benchmark, the CIS Foundation's benchmark, to create a set of dashboards that allow us to monitor all the cloud resources that an organization is using. The entire configuration can be implemented using standard configuration files and deployed with terraform from the command line interface. Why is this important? Because we want to influence accountability, and using this process we can show progress. Real leaders show they don't tell here's what I'd like you to walk away from this section with. X ray is a tool that is used to scan your builds and the specific way that we use x ray reports influences accountability. Our process leverages x ray vulnerability reports in a simple way that can be used by any organization. As one of my mentors always said, you can build a river in a desert. We can make something out of nothing. Peter Thiel zero to one, this is an example of a shift left pipeline. So I asked you, are you doing it the right way? You might be, but here is a perfect example. This code could all go out to production without any proper security. It could be Uber. The code could go out and get hacked and then become the next biggest press release. However, security has been pushed forward in the pipeline. This is a perfect example of shift left. In this case, we have moved the security forward in the pipeline, preventing code from going out to production. Not only that, here's an example of a vulnerability report that can be handed to management with the specific fixes that need to be done. And we can stop these builds before they go out into production, preventing your company from becoming the next Uber. When we are communicating with upper management, we want to provide the proper data to support our progress. It's all about reporting. We can use an x ray report to generate vulnerability reports to show third party vulnerabilities associated with an application, and these reports can be downloaded in a PDF format so that they can be distributed to the right team members. That's the key we're showing. We're not telling. So first, let's be honest. You don't want to do this the hard way, and if you don't have the appropriate numbers and statistics to support your claims, no one's going to believe you. Nobody trusts anyone anymore. And these results are easy to read and they're easy to fix. It's simple. JFrog has the best support in the world. Between JFrog and Veracode, you really can't go wrong. The first step is initializing x ray. We have to configure a scan using a standard Jenkins pipeline script, and this can be done with an x ray plugin. The first step is ensuring that we've configured the proper build info before we launch the analysis on the code. So the specific build info that we want to make sure is set correctly are the arguments for the RT build info function. These include the server id, the URL, username and password. After we've successfully initialized the build, the next step is publishing the build info to the artifactory repo. We want to make sure that the builds are available for team members to have access, and each of these builds should be clearly visible in artifactory repo for a given organization. Then we want to configure a report. One of the most important aspects of this process is generating report. The report provides the specific details on the status of the builds that we're analyzing. They provide the specific steps for remedying the vulnerabilities associated with the application so that developers do not have to waste time trying to figure out how to fix vulnerabilities and management can just look at the report. This provides the perfect situational awareness and context for having a serious conversation about what needs to be done. So why are we doing this? Again, we need to influence team accountability, and using this process we can show progress. Real leaders show they don't tell. We have a strategic application security review that leverages research, proven leadership techniques, and our methods can be implemented in any situation to get individuals to take accountability for their work. If we were able to automate the entire management aspect of these meetings, and we have the data that shows that they lead to positive outcomes. You can build a river in a desert, something from nothing. Here are the results of these questions. As you can see, we've identified a security champion. We've reached a new certification level. We're consistently scanning. We implemented docker container scanning. Let's go down to what's happening next. Vulnerabilities either need to be addressed by mitigation or they need to be fixed and guidance on these vulnerabilities would be provided as necessary. But we're going to stop the new vulnerabilities from being introduced and we can see what was used to get there, why we're doing it, and who is going to be helping to get the work done. This is an example of automated ticketing. It's fully self service. People are taking responsibility for their work and they're not wasting time in meetings. How much time are you spending in meetings? So you have to identify a ticketing system. We need a way to provide auditing and traceability for work that has been completed and the work that needs to be done. Selecting a good ticketing system is the first step in assuring cross team collaboration. It allows everyone to see what each other is working on so that projects are completed and deadlines are met. We want to establish clearly defined goals that show what we are trying to achieve. What has been done a key aspect of this is documentation. It's critical. Without an accurate record of what has been done, we won't know what needs to be changed to improve for the future. And by providing a consistent record of application security reviews, teams will be able to coordinate with each other so that work gets done on time. The primary purpose of application security reviews is to provide a healthy discussion about the status of an organization's security objectives. It's recommended that these reviews take place on a weekly basis so that the team consistently moves forward. They can be scheduled either using Microsoft Teams, Zoom, Google, hangouts. The important aspect is that they are recorded and the meeting minutes are distributed before and after the meeting. When the team meets, everyone should read the previous reviews meeting minutes and they are prepared to answer the specific questions that each member of the team is responsible for answering. And when you start that meeting, you start with a question. Each meeting is recorded, so we need to provide a way for automating this ticketing. That means a fully automated solution for task management. The days of manually going in and assigning tasks are over. This gives teams the flexibility to work on their priorities without having to waste precious time in meetings. Ask yourself, how much time are you spending in meetings? Wouldn't you prefer to spend your time getting the important things done? The added benefit of using automated ticketing is that we still retain the records of what has been done and what needs to be done in the future. Have you ever lost a great leader? There are five key questions that every member is responsible for answering. These are research proven questions that come straight from Bob Moore, CMMC MCC president of effectiveness, Incorporated. The simple program that we've outlined utilizes these questions to take a midsize organization from a nonexistent security program to a veracode level three certified partner with fully automated remediation supported by full scale security information event management monitoring with proper infrastructure as code resources to meet the guidelines as set forth in the center for Internet Security Foundation's benchmark. If you'd like these questions, please message me after the presentation. Why? Why is this important? We are influencing accountability, and using this process we can show progress. Real leaders show they don't tell. All right, let's take a victory lap. As one of my greatest mentors always said, you can build a river in a desert. Here we go. We've got our software composition analysis in Veracode. This platform will allow us to see all the third party vulnerabilities. And then here we have our full scale devsecops pipeline with all the pain shifted to the left forward in the pipeline as opposed to after the pipeline. So that way you don't become the next Uber or equifax or whatever company is being hacked these days. It's very important that you do not let your code go out into production with a bunch of vulnerabilities. Otherwise you will definitely make it into the news. Here is an example of the automated security ticketing that we have set up for application security reviews. This is an example of the ticket with all the details. As you can see, the details here matter because it gives the developer the ability to go fix the vulnerability without any issue, without having to go to a meeting or wasting a security engineer's time. Here's an example of the Veracode score. This Veracode score is an indicator of how well the security is set up in the Veracode platform. From a security perspective, they can use these results to go speak to customers in sales and say we actually take our security seriously. This organization went from a 58, which means extremely hackable, to 94. Here we have an example of the elearning curriculum. As I said, a key aspect of team management is having the effective learning component available for developers. And this is a custom curriculum that was designed based on half a dozen conversations with the development team. Here we have the pre configured filebeat dashboards. As you can see, you can see all of the organization's resources in the cloud. And these dashboards also allow you to monitor the APIs. Again, nobody's hacking this application. And the security engineer has the added benefit that they can actually support their penetration tests with the systematic logging from the security information event management system. You can see that SQL injection is being popped right here. Added benefit. The custom penetration testing workflow utilizes the results from the static application secured scan to execute successful penetration tests. As you can see, this developer was naive enough to actually let SQL injection go into their code. And so the next step here is to take these results and then go pop them with burp suite. Those results that we saw at the beginning with over 254 vulnerabilities from Jackson databine, pretty much gone down to nothing. So use it or lose it. Here's container scanning we can see set up in the veracode platform, right? This is automated. It's scanning all the applications. The security engineer does not have to go in there. What this does is it provides all the results in the platform, one centralized location for developers and security engineers to work together to reach meaningful results and help their organization reach the level of security maturity that they want to be at. This trend here shows that with consistent scanning, what we noticed was the vulnerabilities in the application went down because the results were so overwhelming. This is an example of the container scanning portion of the veracode platform. As you can see, there's over 19 microservices here being monitored. The ones that have Red X's on them mean that they're actually vulnerable. All a developer has to do is go into this platform, click on the service that'll take them to veracode, and then they click on the fix button from the veracode platform. And then they can see the specific instructions on how to fix the vulnerability. It's very simple and it's easy for the developer to schedule these application security reviews. As I said, you could use whatever you like. Simple Google Hangout will do. All you have to do is set it up so it recurs on a weekly basis. This is an example of the Veracode dashboard. As you can see, there's 1675 vulnerabilities remaining. But over 1547 vulnerabilities were closed. And in this organization, all of the high severity to high vulnerabilities were removed immediately. Now, as I said, prepare your minds for a paradigm shift. I'm now going to show you an invisibility cloak for the Internet. Now you see it, now you don't. The icon is spinning here. What you're seeing here is someone accessing the devsecops pipeline that we described. What is happening here is we're taking it off the Internet. So not only does this provide the proper security for all of your code to be pushed to production in a secure fashion, it's also guarded from the Internet. It's wrapped up in an invisibility cloak using a zero trust client. For more information, please reach out to me. So do you want the benefits of implementing this program? Do you want to save time and money? If you want to take your organization to a mature devsecops capability, then here's my contact information. I'm happy to offer you a free sample of content. Also, I'm happy to discuss your specific needs in a separate information session. We will do everything for you. You don't have to spend time worrying about management. You can focus on the more important things. There's never enough time. Thank you for yours. Scan this QR code to access more information related to my contact. To show a token of your gratitude, tweet at Evan Gertis at comp 42. Hashtag success. Hashtag entrepreneurship. Hashtag comp 42. Subscribe to me on YouTube for more content. Again, thank you. I'm humbled by this opportunity. Wouldn't be here if it wasn't for all these great mentors and wonderful people who have helped me out in my life. Here are the references to the research papers that were used. Again, reach out to me. I'm happy to provide you with the slides and the links to these papers. These are a combination of the two most referenced papers on implementing a devsecops pipeline from connected papers. Here are all the resources. These resources provide the getting started guide so anyone can take this program and go do it on their own. This is a continuation of this. So again, please see me after the session. Message me and I'm happy to share these with you. Thank you. There are any questions. I appreciate your time. This has been a wonderful opportunity. My name is Evan Gertis. I'm a Devsecops engineer. I graduated from the University of North Carolina at Chapel Hill with a bachelor's in science and physics and the Allen Paulson College of Engineering at Georgia Southern University with a master's in computer science. I'm a certified terraform associate developer in electrical and computer engineer, and I'm a certified security plus information security professional. Thank you.

Evan Gertis

Founder & CEO @ Inner Growth Training & Services

Evan Gertis's LinkedIn account Evan Gertis's twitter account

Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways