Conf42 Kube Native 2023 - Online

Git Sh*t Done, the GitOps Way

Video size:

Abstract

Are you an avid cloud user, and are now contemplating Kubernetes for your project? Did you already start to play with clusters, controllers, pods and anything K8s? In both cases, stop what you are doing right now, and let me help you in your container orchestration journey by introducing you to GitOps. I know, I know… you barely jumped the DevOps wagon, and there is already this new kid in town. Worry not my friend, because this talk will set you on the path to success for implementing Continuous Deployment (CD) for cloud native applications. You’ll learn what the hell is GitOps, and how it will help you better collaborate with your team, deploy & iterate faster within a more secure environment, while saving some costs on the road. In the end, you’ll leave with concrete steps to implement best practices to get shit done, the GitOps way!

Summary

  • Frederic Harper is a principal developer advocate at Cube first. He will talk to you about getting stuff done the GitHub's way. When you watch the video, please feel free to ping me on Twitter.
  • Gitops really help you to manage cloud native systems with Kubernetes. Now we think about infrastructure as code. We're splitting the CI and CD pipeline. Instead of having the git repository that push things, the CD part will grab information, and it becomes your source of truth.
  • Cube uses a GitHub repository structure that we're using at Cube. It's the source of truth of my Kubernetes cluster right now. And we manage different secrets using Ashicorp vault that are all connected together to the application. I'm going to share a little more about the pros and cons of using that method.
  • The GitHubs principle is that everything should be in the git repository. Software agents automatically pull the desired state declaration from the source. From my point of view it really brings greater collaboration, auditability and simplification of access control. There's some, maybe not costs, but you may see some part of it that may not be as beneficial.
  • Argo CD is a tool that helps you manage your Githubs and your CD pipeline. The demo shows how it works on a local clusters on key 3d right now. The applications would be exactly here. But flappy is being out of sync. But after a couple of seconds everything should be good.
  • The open GitHubs slash project is a good place to start to learn more about the GitHubs principle. We have an open source product that is free. You can also, if you don't want to go on the cloud right now, pay for resources to learn about Githubs.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello lovely people. My name is Frederic Harper. I'm a principal developer advocate at Cube first and today I'm going to talk to you about get shit done the GitHub's way during the talk. When you watch the video, please feel free to ping me on Twitter. I'm at Fr or share screenshot or things that I've said that youll agree or disagree with. I'm all about discussing stuff and I don't have the only truth, so please connect with me. I'm going to be more than happy to talk to you. So with that said, let's jump on the talk. We only have 30 minutes together today, so that's going to be a high level overview. I'm going to try to put as much content as possible with a little bit of demo so you can see how it's working and how you can apply this to your day to day work. So let's start with maybe something that you already know, something that you're comfortable with, which is DevOps. So we are used to DevOps, or maybe you're starting to learn about DevOps and there's that thing called Gitops that is there since a little bit now, but still something a concrete that is quite new for most people starting or even people that are in their actual cloud native journey. So when we think about Gitops, actually, sorry, when we think about DevOps, we put the application code in git repository most of the time. And we're going to have that CI CD pipeline who's going to basically validate our code and do everything we need and deploy our code to the cloud to on premises. Actually that delivery pipeline will put your application somewhere. So this is the kind of process that we are used to when it comes to DevOps. The difference now is that you can also do that with DevOps. But it's an important part of Githubs is that now we think about infrastructure as code. So I'm a big fan of infrastructure as code. Hiac, one of the most popular out there is terraform. That helps you to do that. And it's basically as it says, like it's having your infrastructure as code. So youll write down some code that will create your infrastructure and you still put this in your git repository. And traditionally more DevOps way of doing things. You're still going to put your configuration sometimes in your git repository. So it's kind of like the industry default way of doing things. We are familiar with that. It comes easy now because we've been doing that for a couple of years. And the thing is that there is some flexibility of doing those because it's a principle that is not specific just for cloud native application. You can use that in different type of environment. But now what we are changing when we move toward GitHubs is that we're splitting the CI and CD pipeline. So it's still there, you still have continuous integration, you can still do whatever you were doing previously in that part of the pipeline. But there's also another difference now, instead of having the git repository that push things, the CI actually the CD part will grab information, will pull the information from the git repository, and it becomes your source of truth. And I will talk a little more about the difference why it became the source of truth and why it may not have been. When it comes to DevOps or using DevOps as your way to manage your application deployment, the things that may be something that may not be useful for you when it comes to GitHubs is that it's really specific for Kubernetes, because on those pods you're going to have mechanism that will grab the information from your git repository to always ensure that you have the latest version on what is in your source of truth, the git repository. So it's Gitops in a nutshell. And Gitops really help you to manage, as I said, cloud native systems with Kubernetes, but in a declarative way, leveraging git. So it's a tool you already know and you already use. You can see this as a branch of DevOps. It's really taking its best practices. But with the big and cases of having git, youll source of truth and not the server. So there are some pro and cons about that. So let me show you a little bit about a GitHub repository structure that we're using at Cube first. So let me switch my screen here and hopefully right now you should see my GitHub repository. So that was created when I used the Kubefirst CLI, which is our product, but I'm not here to talk about it, just to show you one example of how your GitHubs repository could be structured. So we have our GitHubs repository here, we have our registry. Cube first is just because I call my cluster like that. And you're going to see there is those yaml file that you may recognize. Those are basically Kubernetes declaration that I have in my Githubs repository that's going to be read by argo CD so we use our go CD in our ecosystem to install update deploy application on my cluster. So I have my ML file here we are using the appsup apps patterns and here you have different components for different parts of application that are installed in my cluster. So my GitHub repository is my source of truth. If I go back here, I have the terraform folder which is my infrastructure as code here. So in my case there is kind of like three parts. So there's a part where our platform lets you manage youll different git repository that are associated with your cluster. So in here if I go and repose tf I have two git repository that were created for me when I created my cluster and we use a terraform provider to manage and create those. If I go back here, I'm also managing my users directly from the git repo. And you may say like Fred, it's starting to be a little bit weird, but bear with me, I'm going to share a little more about the pros and cons of using that method. So here let's say I have a developers terraform file and I could have the developer one user directly by doing a commit on my git repository. And also there's a third part for us where we manage different secrets using Ashicorp vault that are all connected together to the application that are pre installed on your cluster. So that's just one way of having or defining the architecture of your githubs repository. This is what we've done. It's a culmination of a couple of years of experience and also based on the needs that we have to build our platform. So everything is there. It's the source of truth of my Kubernetes cluster right now. So let's go back to the slides and about that now. Okay, I introducing you a little bit about Githubs. Let me talk to you about the GitHubs principle. So I show you a little bit how it looks like compared to DevOps, but there's some principles. So Githubs, it's not a standard per se, but there is the open Gitops group that is working on some principle and usually people, when we talk about Gitops we try to follow those principles. So the first one is that it needs to be declarative. A system that is managed by GitHubs must have its desire state expressed declaratively. So everything should be in the git repository. It's kid of like another way to say that it should be version and immutable I don't know why I struggle with that word in English. So the desired state is basically stored in a way that enforce immutability. So the servers are never modified after the deploy in the sense that you don't go yourself there and modify stuff. It's version because you're using git and because you're using git, you retain a complete version history. So that's kind of like one of the benefit of GitHubs. It's pull automatically so you don't have to push information. It's really, there is software agents that automatically pull the desired state declaration from the source, and there's benefits for that. And there's some, maybe not costs, but you may see some part of it that may not be as beneficial. So I really want that talk, to be honest about GitHub. So I'm going to come back just right after about the pro and costs of using the GitHubs principle, or actually thinking about using GitHubs when going through your cloud native journey. And the fourth principle is continuously reconcile. So this is also one of the pros is that the software agents, because they continuous pull the information, they observe the actual state of your cluster versus the desire state that is in your git repository. So it would always attempt to apply the desired state. Hence the source of truth is being git. So there's the benefit about that, that I'm going to talk right now. So the benefits, when it comes about Githubs, it's really you're using familiar tools and processes, git, git workflows, those are the things that you hopefully already know and use day to day at work. There's also the benefits of infrastructure as code. Youll have the change history because of the file versioning, it's easy to revert. And there is the code change revision processes. In the best world scenario, you should always never commit directly on the repository. You should do a pr and someone from your team, at least one person from your team should review the changes. So that's really good for accountability. But also that's really good to be sure that you're not making some mistakes, because there is someone else who will look at your change. There's faster deployment and iteration, and it's easily reproducible because you have the secret source. Everything is in your git repository. There's also no cluster drift, which means difference over time because of server changes. So when we're doing DevOps, like maybe you had no clusters drift because everybody was already using your git repository as the source of truth. But if we want to be honest it's rarely the case. So sometimes there's an issue on the server side. There is an issue in something in pods and what are you going to do? You're going to go in the prod server and you're going to connect with SSH and you're going to try to fix things and that's going to work and it's going to be beautiful and you're going to forget about it. So you have a change that is important for production, that is only on the production server. But now because everything goes through youll Githubs repository, everything is there. So there's nothing on the server. So you're not losing changes. There is no cluster drift as I said. From my point of view it really brings greater collaboration, auditability and simplification of access control. Again always coming back about the source of truth. I see this as more secure because there is no need for individual infrastructure access. There's also no need for your CD pipeline to have access to your Kubernetes clusters because it's the other way around. We're not pushing things, we're pulling things inside the cluster. There is more transparency because all the git manipulation and as I said, these agents are running inside the cluster so no need for the CD to have access to the server. When we think about Githubs too, there's the cost saving because there's the automation part versus the manual execution, deployment testing, easily reverting or rollback some changes so it's prone to less error. And as I said, it's also easily revert like you can easily reproduce and revert also I would argue that there is some cost saving there and it's also a great approach for a simple stateless disaster recovery model because there is no server state. There is only like the source of truth state. If I want to be honest there are some challenges or maybe we could even say some cons when it cases to Gitops. So the learning curve can be a little bit steep. Especially if you have no knowledge of Git, if you have no real knowledge of infrastructure as code or some of the best practices, or maybe you're not even totally familiar with DevOps approach. What I see as a benefit, some people could see this as like a challenger accounts because there's more layer of approval because everything goes through git. So in cherry, as I said before, if my team was there they would be like no Fred, don't like when you commit directly on the repo. Sometimes we do this in demo and live stream and I'm always complaining and I say that I'm going to do that in my demo right now just for the sake of time. But like in a day to day in production, when you work on your product, you should not do that. So you do a pr, someone will review your changes at all level, on the infrastructure side, on the application side, or everything you need to run your cluster. So it means more approval, but it also brings all the benefits that I said before. It can be a little bit harder to customize the process. It's kubernetes centric, as I was saying before, and things can go wrong between the source of truth and the CD action. So it depends on the tech you use. And you know what, lastly, it's not magic. It doesn't fix bad practices. So I'm sorry about that. I know you were maybe hoping for some magic solution. When it comes to Githubs, it's not the case. So obviously it's just a tool. But I firmly believe that it's a tool that really helps you to have a better cloud native journey and ensure that your processes in place are helping you moving forward and be successful in your endeavor. So let me show you a little bit more how this is exactly working. So let me go back to my browser here and you're going to see here I'm in ergo CD, so it's fine if you don't really know Argo CD. It's one of the tools that we really like out there. There's like Flux CD. There's other tools out there that help you to manage your githubs and your CD pipeline when it comes to managing githubs. But we decided to use argu. This is the tool we town, we liked it. And if I go here and I search for an application called Flappy, it does not exist and it's normal because it does not exist. So what I'm going to do here, I'm going to go in my registry folder, I'm going to go here and I'm going to say I want to add new file. So had file create new file, call this flappy Yaml. And I'm going to be lazy. I have my shortcut here. As you can see, it's a YAml declaration for an application called Flappy Bird that's going to take a chart and install a version of it's basically flappy bird. We rename it to Flappy Q array. And I have my ingress route here because I want to be able to access it with flappykara Q first dev. So if you want to try this at home I can share your link if you share with me but that link won't work because I'm doing this on a local clusters on key 3d right now. So I'm going to do the unthinkable. I'm going to commit directly on the main branch. So had flappy, flappy k ray and I'm going to commit directly here, I'm going to save that and what I'm going to go back here so I'm going to refresh CD, I'm going to go actually youll should have probably check in the registry before so I'm going to be sure to sync. So it should sync automatically. I think the default is streaming it, I never remember. So if I synchronize that I should see at some point here, oh registry, something is working here. I should see my flappy key ray here. It's out of sync so it's syncing right now and if I give it a little bit of time. So what's happening here? The applications would be exactly here. So what's happening here is that flappy is being out of sync. So youll know where I was thinking about trying to get always the latest state. So Hargo CD checked my git repository, realized that the application was deleted. It's not something that should happen. So sometimes Hargo CD is doing some weird stuff but the application will come back right away. So if I just fire sync again that should be back and we should see flappy key ray. I don't know how to. Yeah, flappy here. Something's going to happen here in a second. It's going to be here. There we go, flappy now let's go see. So it should be in the process of being deploying, going to refresh. That still happening. It's out of sync. And what I can go here, I can check the diff and you're going to see that there's a difference. So it's not deployed yet. There's a lot of difference. But the job of ergo CD, as I was starting to explain, is that it will check on my git repository, it will look at the information that is there, it will compare to what's deployed on my cluster and it should update the registry. Oh, what happened here? So it's still syncing and we should see something. So what's going to happen now? Let me just go back to the presentation part and we'll come back to. Oh, actually I think it's good now. I think he was able to deploy. If I go here and I access Flappy Kray, it's a page not found. So what's happening here? I don't know what's happening here. Let me check what's happening. Maybe I made a mistake with my yaml file. So it's sync. Oh, I know what's happening. I know what's happening. This should be good. The chart version, flappy Kara. If I check the certificate, should be good. That's out of sync. There we go. Not sure what happened here. So now it's syncing, it's installing. Going to give it a couple of seconds and everything should be good now. Oh my God. So little live demo problem. And now I have my kray application that's been installed from my argo, actually from my GitHub server. I shouldn't play and try to explain something because it's obviously not working. I cannot do the same two things at the same time. So let me go back to my talk. So youll see, you see, I install the application by heading a Yaml file and argo CD. Fetch it from the GitHub steps repository, install it, deploy it with one small hiccup because that was a quote unquote live demo. So some resources if you're starting your Githubs journey, there is the open GitHubs slash project. This is the open GitHub project or working group that I was talking before. So really a good place to start to learn more about the GitHubs principle or to just to get more information about it. A great repository from weaveworks. It's awesome GitHub. So if you know the awesome repository that you usually have on GitHub, there are a bunch of awesome resources about specific topics. So this one is a pretty good one too. And youll obviously I need to say it if you want to replicate and start to play with Githubs, we have an open source product that is free. It's available@GitHub.com cube first slash Cube first, because the name of the project is the name of the company and youll can try it on AWS Google Cloud, which is actually in beta right now. But like AWS and Sivo, they are production ready and we have support for Google Cloud. Digital vulture, they're in beta right now, but they're going to get out to beta really soon. And you can also, if you don't want to go on the cloud right now and pay for resources to learn about Githubs. We have the key 3d experience that you can also use locally. You just need to have Docker, desktop, everything is in the documentation, so feel free to use that. So I forgot to remove that slide. It's a recording of the talk. It's not a live conference. But if you have any question, don't stop there. It's not because you're watching a recording of that talk that you need to stop there so youll can go on Qfirst IO slack. We have a GitHubs channel and you don't even have to use our product. It's just like, if you have questions about Githubs, if you have question about the things that I've said during my talk, youll go there. We're just passionate people about the Githubs principle. We're going to help you again, even if you don't use youll product. We're just that kind of friendly people. And also, if you don't want to go in the Slack channel because I don't know, you're afraid that I'm going to try to sell you how to use our product, you can always contact me, send me an email, fred at kubefirst IO, ping me on Twitter, I spend some time there. Or you can even connect with me on LinkedIn. And if you prefer face to face, unless you live in Montreal, in Canada, where I live, where if it's the case, I would be more than happy to go for coffee with you. But if you don't, we can have like virtual coffee chat. So you go on Fred devcoffee, and you're going to have access to my calendar and you can book a free 30 minutes discussion with me. We can talk about Githubs. We can talk about anything you want. I'm just a social person, and if there is a way for you, for me actually to help you, I'm going to be more than happy to do that. So on that note, I hope it helps you to learn a little more about Githubs. And hopefully that's going to be, and hopefully that's going to be a principle that you will be able to use in your day to day at work. So on that note, have a good rest of the conference.
...

Frederic Harper

Principal Developer Advocate @ Kubefirst

Frederic Harper's LinkedIn account Frederic Harper's twitter account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways