All right, let's take a look at what Gitops is with Argocd, and then we'll spend some time understanding vault and the injector in kubernetes. Then we're going to follow that up with a demo. So, intro to Gitops with Argocd if you're not familiar with Gitops, it's the process of having your application code and your infrastructure code all stored in git. And basically git becomes your source of truth and you deploy straight from git. In this case, Argocd is a tool that implements Gitops principles and it's implemented as a Kubernetes controller which continuously monitors running applications in your environment. It compares the current live state against the desired target state as specified inside of your Git repo. A deployed application whose live state starts to deviate from the target state is considered out of sync. You have the option to have auto sync enabled, so revert back to what you want your desired state to be, which is in your Git repository. And again, that's a source of truth that is available, or you can leave it at the default of manual and then you can alert on a deviation if you wish. So ArgoCD reports and visualizes the differences while providing facilities to automatically or manually sync the live state back to desired target state, as I mentioned. And then any modifications made to the desired target state in the Git repo will automatically get applied and reflected in your target environment. So it's an amazing technology, and many organizations are moving to Gitops. Now let's take a look at the app, the demo app that we're going to use today. I call it the school app. And as you might know, I offer an academy, a teaching academy for DevOps and platform engineering. And as such I use this school app as a demo application in my tech and aid academy. So here what we're going to do, we're going to have this application that allows users to enroll in courses. And you can see the back end here, the course that was created and what users or what students have been enrolled into that course. And the components of this application is basically it's written in view JS. That's the front end JavaScript framework that is used for the user to interact with the front end. The backend API is written in fast API, which is a great python API framework. So as the user interacts with the front end, the front end talks to the backend app, and then our data layer is using MongoDB, that's our database, and then our API talks to the database and of course the database responds back and then the data is rendered back to the user in their browser. Now, in the very beginning of our demo, we're going to hard code the connection to MongoDB inside of our app. So the credentials, the username and password, they're going to be hard coded in here, which is definitely not the way to go in production. You need to have some sort of a method to automatically rotate secrets and store those secrets outside the application. In a secrets management store such as Hashicorp Vault, the application output in Kubernetes will look something like this. When you run Kubectl getall, you're going to see that we have three pods. We've got an API pod, a front end pod and the MongoDB pod. They're all deployed as deployments. And of course we have services for each one of those pods and they're using cluster IP for their service type. All right, let's now add vault to the mix and see how things will work for our school app application. So once again, we have the same workflow. We got our user talking to the front end and we've got our fast app API and we got the front end talking to the API. And then MongoDB needs to be accessed. But this time we're going to rely on vault to supply us with the necessary credentials. So the API will reach out to vault, it needs to authenticate into vault and then reach out to get the database credentials so that the API can now talk to the database. And of course the database talks back to the API and the front end, the application is rendered in the front end. So to get to that step, we need to understand first, the first step of accessing vault and in our unique case here is going to be using Kubernetes authentication method. So the Kubernetes authentication method is one of many authentication methods that vault allows. And in this case it is a method to solve the secret zero problem. Basically what the secret zero problem is, how do you access the first secret, how do you get secrets out of vault? But first you need to authenticate into vault. So how do we authenticate into vault securely? So that's where we have a vault agent sidecar that can sit beside your application container in the same pod. This pod is going to have a service account or youll application is going to have a namespace and a service account. And what will happen is the application pod will get, or the vault agent will get a jot token from the kubelet inside the node where the application is running and that jot token is going to get passed over as the vault agent authenticates into vault. And there is a connection between the vault server and the Kubernetes API that we'll set up. And that way the vault server can pass that jot token over to the API and basically ask the question does the pod or did the pod that I just used the jot token for, was that initiated under your management, Mr. Kubernetes? Kubernetes will come back and say yes, I know about it, it's under my management and that is the authentication piece. Then what will happen is vault will come back, deliver a token and a token has a policy attached to it to authorize the application or the vault agent to retrieve secrets at specific paths. So that's kind of the Kubernetes authentication method in a nutshell. And from here we want to talk a little bit about the vault agent sidecar injector. So the vault agent workflow in Kubernetes looks like this. You have an application Kubernetes pod with a mounted volume inside of this pod. We've got our application. So our app lives here. We also have the vault agent sidecar that I just mentioned and we have our vault server. So what happens is the vault agent will go ahead and authenticate with the vault server. The vault server will respond back with an application token. The vault agent will store this app token inside of a sync file in the mounted volume so the application can actually access this token. And if the application is vault aware we can stop here and the application can grab this token, talk directly to the vault server and retrieve the secrets directly. But that of course means that youll application code needs to have a way to talk to the vault API and retrieve secrets. So you're going to have to make changes to the code. But we can take this a step further and we can actually say, you know what, let's have the vault agent retrieve secrets on behalf of the application and that way it makes the application vault unaware and you don't have to make any changes to the application. Of course vault needs to check against the policy that's associated with that token to make sure that the vault agent can retrieve secrets from the server at the particular path that it requested it at. And from there there's something called the vault agent template where we can actually templatize how we want to render those secrets for our application. So what happens is now the vault agent is going to use this template to grab the secrets and render them in a particular format to allow the application to retrieve my username and password for my MongoDB. And once the application can read this out of the file that's again in the mounted volume here, it can now connect to the MongoDB database and the application continues to run as before. So let's talk a little bit about the Vault agent sidecar injector. What it is it really alters the pod specifications to include the vault agent containers that render vault secrets to a shared memory volume. Using vault agent templates. The app containers within this pod can consume vault secrets from the shared volume without being vault aware. As we saw in the workflow, the injector uses or is a Kubernetes mutation webhook controller and it works by intercepting pod create and update events in Kubernetes and then the controller parses the event and looks for the metadata annotation. Hashicorp.com agent inject is true and if found, the controller will alter the pod specification based on other annotations that are present so mutation effects. Every container in the pod will be configured to mount a shared memory volume. This volume is mounted to the path vault secrets by default. Of course that's configurable and will be used by the vault agent containers for sharing secrets with the other containers in the pod. There are two types of vault agent containers we can inject. There's the init and the sidecar, and the init container will pre populate the shared memory volume with the requested secrets prior to the containers starting. And these are very good for cron jobs. So if ever I'm running a backup job, for example using cron, I will use the init container for that. And then the second one is the sidecar container which will continue to authenticate and render secrets to the same location as the pod runs. And this one's really good for long lived containers. And if you use dynamic secrets later, which I highly recommend you do, as those secrets change over time, the sidecar container can render those secrets for the application dynamically and then using annotations. Init and sidecar containers can also be disabled so vault agent configs to render the secret there are two methods of configuring the vault agent containers to render secrets. The first one is using annotation agent inject secret annotation the second method is to use a configuration map so config map that allows that contains the vault agent config files and only one of these two methods may be used at any time. The config map can provide more details for configuration beyond the annotations. So what's an example look like? This is what we're going to do in our demo. Here's our annotations and you can see here that agent inject is true. Agent inject token is true configures vault agent to share the vault token with other containers in the pod. This is helpful for Vault aware apps that need to communicate directly with vault but require auto authentication provided by vault agent. So if you're just going to use the token for a vault aware app, you need to have this. The update blocks further mutations by adding the value injected to the pod after successful mutation. And here's where we render the vault token in this particular path. App secrets and under the file called Token and we'll see it in the demo. And the role we're using here is called school app. If you want a full list of annotations, you can go to this particular URL to see all the available options for you. Let's take a look at the vault agent template. Quick review here. Vault agent uses the console template project to render secrets. So the console template is a project that was actually used with Hashicorp console that got extended over to the vault agent to be able to render these secrets as a template. And then this is useful for vault unaware apps. As we mentioned before, the app's file system has the secrets dropped in by vault. The app doesn't talk to vault directly and all it needs to know is what files to find those secrets from and how to read them. And just again, a quick overview or quick review I should say, of this workflow. Once again we have the mounted volume where application lives. The vault agent lives here in the same pod. It talks to the vault server by first authenticating using the Kubernetes authentication method. The vault server responds with a token. That token gets stored in the sync file and we can stop here. As I mentioned before, if the application is vault aware, can retrieve the token from here and talk to vault directly. Or we can take this a step further and get the vault agent to retrieve secrets on behalf of the application and store these secrets in a particular file that is based on a template so we can get the particular format that we want for our application to talk to whatever it needs to talk to, in this case the MongoDB database. And now we're good to connect. What does this look like in terms of more annotations? So now we stopped here last time at this line here, but now we want to take it to the next level and add the templating for our application. So as youll can see here, we are adding this annotation agent inject secret school app MongoDB username and this is the path inside of vault at this MongoDB path inside of internal data school app. And then we also have the password file. So these two will effectively tie into two files that you'll see down here. Down here you see another annotation for our template and this is what the template looks like. So all we're going to do is just render the school app DB username right here in between here. And this is using Golang's templating language. So we're grabbing the MongoDB username from this path inside of vault and also doing the same thing for the password in a separate file. So we'll see this inside the demo, inside our container. You'll see that we have two files that will get rendered app secrets school app MongoDB username and school app is going to be the username and then app secrets school app MongoDB password. The password is going to be Mongo root pass that's going to get rendered in here and that's pretty much it. From a slides perspective what we're going to do, let's now dive into a demo together. All right, let's now have some fun with our demo. So I've got this all set up for you in GitHub and you can use code spaces to get this all set up without having to do anything on your local machine. So if you go to our repository here, the GitHub repository, first thing you want to do is fork it into your own environment. So make sure you go ahead and fork it. And then once you fork it, you can run a code space on main like this. And I already set up the necessary dev containers that will set up a mini cube Kubernetes cluster. It will set up really everything that's needed in our environment. So the one thing I will note though is for this to work, you're going to need to have this run in your own visual studio code on your desktop. And I'll tell you why in just a minute. So go down to the bottom here where it says code space at the bottom here and select open in vs code desktop. What this will do is it will open this particular code space remotely inside of your visual studio code installation on youll machine. So I'm running this in Windows here. It doesn't matter what operating system you're running it in, it's going to get started. And you can see at the bottom it says opening remote. So it really is talking to code space, which is GitHub's own cloud, where you can have development environments in there. And I use it quite extensively in my courses, makes things much easier for my students, so they don't have to worry about binaries and dependencies and things breaking. If it works in my code space, it will work in your code space, that kind of thing. All right, now the reason why I'm asking you to run this in your own visual studio desktop is because when it comes to the communication between the front end, the back end, the database, we're running all on local host. If you were to try to run this in codespace, Codespace gives a port. If you go to ports here, it'll give a port for each one of the exposed ports. It will have a particular host theme, and my front end won't know how to reach the API, and the API won't know how to reach the database and so on. So because this is a demo, I want everything to run on local host. So you'll need to do this in your visual studio code desktop. So while I was chatting here, our minicube cluster should be running right now. I have a bunch of aliases, if you notice, in here. In my startup script for this code space, I use a bunch of aliases for kubernetes commands to make my life easier. The most famous one is KGA, which Kubectl get, all kdell for kubectl delete and a bunch of others kg for Kubectl get. So I kind of render this here to make my life easier. I also have k nine s installed here as well, if you like that. And I also have KubectX and Kubernetes installed, which is really good for me when I want to work with namespaces. So let's render this on the left here so we can see a bit better. And I'll go ahead and pull up my readme file, which I have all my notes here, and we start to look at the demo steps one by one over here. So let's look at KubectX. We'll see that we have only one context minicube. If youll run Kubernetes, that will show us we've got four namespaces, just the default mini cube cluster. So first things first, let's install the school app, and I'm going to run a bunch of commands here to get this going? I'll explain this of course we're creating a new namespace called School app. We're creating a helm repo well adding a helm repo for Bitnami, we're creating the MongoDB Helm chart and then I'm going to create my application here I've got my helm chart. I've exposed my application or packaged my application as a helm chart, called it school app. And from here we can quickly take a look at the new namespace called School app Run KGA. And I see that MongoDB is installed here. Now I'm going to install my front end and use helm install front end to install my front end. I'm going to use helm install my API to install my API. And now if I run kubectl get all I see that I am running my containers here. They're not quite up yet, so we can do a watch kubectl get pods and just wait until my containers are up and running. And they are. I can see API front end and school app mongo. So as I showed you in the slides, here is my school app ready to go up and running. Just out of curiosity, if you want to see what this looks like, let's expose our front end and our back end. So that's the front end on port 8001, the back end on port 5000. I can open these in a browser window. And here's our fast API running. If I go to the docs URL I can see all my endpoints and if I'm interested now of course to see my front end, I go to 8001. That will bring me to my front end of the application. I can go to the admin side of things. I can create a course or add a course. Here's my vault 201 course which a plugin here. All what I'm showing you today is from this course vault for applications in Kubernetes. If you're interested in that, of course you can check that out on my technate academy site. Now we can also see the front here, front end for the application for the users, they can enroll into a course, enroll here. And now this user is enrolled. If I go back to my admin I can see this user is enrolled here and I can delete as well. So just showing you how I'm interacting with my database and storing things in the database, deleting things from the database, everything is running very well. And however though everything is running with credentials that are hard coded into the application. Meaning that if I go in and let's see here. If I create a new tab and I open my, let's see my API logs. If I run this command, it will show me the API logs that we are looking for here that we were running. And if I get this correct, here we go. You can see that MongoDB credentials using hard coded values that appear in GitLab, which is not a good thing. Basically these are hard coded into the application, they're pushed into git, and they're visible for anybody who has access to the git repository. This is a big no no. We don't want to do that. So we're using to introduce vault next. Okay, so let's get out of here and let's see where we're at. We are going to now install Argocd. Okay, so what we'll do is we're going to install the ArgoCd server. So let's create a new namespace, call it Argocd. All right, and we're going to use this Kubectl apply command to install Argocd for us in that particular namespace. Let's change our namespace to Argocd. And let's watch our pods to make sure that argocd is up and running. So we can see here a bunch of pods that are still in the process of coming up. Almost there. We just have a couple more that need to be ready before we go and we're done. Perfect. That was quick. All right, now let's expose the ArgoCD API server so we can actually see the UI so that we are exposing now port 8002. Let's open that in our browser. And from here we should be able to access Argocd. We just need our username and password. So let's go ahead and grab that. By default, ArgocD puts the password in here in this secret. So let's grab it. And there it is. Let's copy that and log in with admin and sign in this way. All right, we have a freshly installed Argo CD server. No applications ready or no applications deployed just yet. All right, so let's go ahead and deploy our application. But first let's go into our school app and delete this namespace altogether because remember we now want to use Gitops as our approach to deploying this application. So let's run Kubectl or kubectl delete or kdelete namespace cool app to delete this namespace. And we'll just double check, make sure everything is gone before we proceed because obviously again we want to use ArgocD as the tool, our Gitops tool, to spin up our application so it can monitor this application and make sure there's no drift. All right, let's run Kubectl, get namespace. Our school app namespace is gone. And now what we're going to do is we're going to run Kubectl apply and use this ArgocD school app manifest to render our application. If you want to see what this manifest looks like, it's pretty straightforward. It's an application for Argocd. It's going to be in the namespace Argocd. The name is school app. The destination namespace is school app. The server is our Kubernetes server. The path is school app subchart, which is right here. This is what we're rendering and we are grabbing the repo, which is our repo that we're in right now, the ArgoCD vault repo. And we are creating a namespace as well. So if I were to run Kubectl gate namespace, I don't see anything yet. It should show up. So let's take a look and see what happened in the UI. There it is. So all I need to do is just click sync. And now our application is progressing, it's syncing. As you can see here we have all the different pieces are running and there is an issue here with our API pod. And the reason for that is before recording I forgot to change my values for my application here to I need to disable these two fields and then we'll come back and fix it later. So let's go ahead and disable this. All right, so true to Gitops, I'm making a change in here. Okay. And now what I'm going to do is I'm going to push this back to my git repo and argocd is going to notice the change and go apply the change. Okay, so we've got status disabled, kind is disabled. Let's go ahead and commit this. Let's just call it start fresh commit and let's sync our changes. And now that is synced, let's go ahead and go back into our ArgoCD dashboard. I'm going to click refresh just to get this to move faster. Otherwise we'd have to wait I think two minutes or five minutes by default. So we click refresh. You see out of sync already. And why is this out of sync? You can click apptif and it will show you what the difference is. You can see the annotations are going to get removed, for example, and our vault status here, this is going to be disabled, meaning that we're removing vault from the equation. And this is what we should have started with. But it's actually a good exercise to show you Gitops in action. So now we can go ahead and click sync and we're going to synchronize everything. And this should fix our API deployment and it should be healthy after this works. So let's see here. So the replica set deleted our pod and see if it will create a new API pod for us. Just give it a few seconds here, maybe even a quick refresh, because I'm impatient. And there we go. Yeah. So here we go. We've got our API pod running. If I go back in visual studio code and run get namespace, I see my school app here. So cuben school app and run qCtl, get everything. My API front end, school app, everything is running. If I go back and just stop the connection and secrets the connection to my front end and my back end, I should be good to see my application once again. So if I refresh that, I see my application once again, I can add courses, I can remove courses, I can interact with my database, no problem. Okay, now remember, we're still in the phase where we haven't introduced vault yet. All we've done is we've introduced Gitops through Argocd. So with Argocd, we can monitor the desired state in our git repo versus the active state, which is what we see here. All is good, everything is healthy. Next step is to introduce vault, which is our final step for our demo today. All right, so let's go ahead and go back to my guide to see where we left off. All right, so we've deployed the app with Argocd now, and we saw that it's using hard code secrets from before. Okay, now we're using to install the vault server in our environment using our helm chart. So let's create a new namespace, call it vault. Let's introduce or add our repo, our helm release here, or our helm repository. And then we're going to install our vault server inside of the namespace vault. And let's see what we have here. We've got containers that are starting. I can see the vault container. I can see the vault agent injector, which is what is the mutation webhook that looks for annotations and goes ahead and creates an agent sidecar for us. So let's watch this until everything is ready. Okay, we're running. Notice that the vault node or the vault server is not using to go into the ready state. It's running, but it's not going to go into the ready state because we need to initialize an unseal vault. So to do that, first of all, we need to exec into our container. So let's do that. So now we're inside the vault container and we need to run a few commands here. First off, we need to initialize vault using the vault operator command. And just notice that this is a demo environment. Please don't do this in a production environment. Production environment. You'll need to do a whole lot of things when you initializing vault. And by the way, vault doesn't need to live inside of a Kubernetes cluster. Your vault server can live somewhere else. But you will need the injector inside of the cluster so it can listen to annotations or see annotations and then inject the agent. All right, so this is important. The unseal key and the root token. We need those. So we will save these in a notepad file. So I save them here in this notepad file. I'll make it bigger. All right, we're going to come back and use these very shortly. Actually right now. Vault operator unseal and we need that unseal key. So this is what allows us to unseal our vault cluster. And now our sealed status is false means that we are unsealed, which is great. Now let's go ahead and exit this container. Okay. Actually one thing we want to do is I want to make sure that I add my authentication, kubernetes authentication into vault. So we actually need to stay inside the vault container to do that. So let's go ahead and go back. All right, so we're back in the vault container and we're going to do a couple of things. You need to export the address, the vault address to port 8200. And we need to log in. So vault login. And you need that root token that we just saved. And if you have spent more than 5 seconds with me on any of the vault tutorials I present, you'll quickly hear me say that the root token is only for creating authentication methods and it is to be revoked in production environments. Of course, this is a demo, so we're going to play with the root token. We won't bother doing anything else. Now here we have our first secret that we want to create. First of all, we got to create a secrets engine. This is the KV version two secrets engine. We first enable it and then we're going to put our secrets in here. Remember our MongoDB username and password? So here's our username and here's our password. And if you want to retrieve that you can run this command and it will come back retrieve your username and password right here. Now we need to create a school application policy. If you recall when I talked about the token and the policy, you need to have a policy in place that allows the holder of the token to access particular locations. So you see here the path internal data school app MongoDB is the same path that we wrote the secret into. So we need to have this policy in place so that the vault agent will have this policy as part of its token when it goes out to authenticate against vault. So now let's set up the Kubernetes authentication method in vault. This allows our vault agent to talk to vault. So this is just a bunch of config that you'll have to create really quickly off Kubernetes config the issuer, the token reviewer the Kubernetes host the Kubernetes Ca Cert we're writing or binding to this service account and this namespace called School app. Very important policies is app which we wrote up here. So this policy is attached to this role that we're creating, the role called school app and we're giving it time to live 5 seconds and maximum ttl of 20 seconds. But basically this is our way of getting kubernetes or vault set up with kubernetes so vault can authenticate against Kubernetes and figure out if any of the resources or pods that Kubernetes spins up belong to the Kubernetes API and that allows vault to authenticate applications into Kubernetes. All right, let's now exit the container. I think we're done with all the configuration for vault. And now let's test the school app with the vault injector. And how are we going to do that? I have this all set up with templates, helm templates and so on. So all we need to do is go into the values Yaml file here as part of the subchart. And I need to just make changes here to do this. Instead of vault being disabled here I want vault to have status of KV, static injector, template file and kind injector. And that basically will allow us to use vault to render secrets. Now I'm going to say use vault for secrets. I'm going to commit this and I'm going to push that. And again, in a production environment, you won't just have the ability to commit and push straight to the main branch, of course. You create a pull request or merge request, you have a peer review and then that gets merged into your main branch if that's the strategy you're taking. All right, so that gets pushed through again, I'm going to hit a quick refresh inside of argo CD and we can see now that we're out of sync, right, so we need to sync this. But before we sync, let's again look at the diff. And as you notice here now we have a bunch of annotations. And if you recall I told you that you need these annotations for the mutation webhook, which is the agent injector, the vault agent injector. Once it sees these annotations it's going to mutate the pod specification and that's how we're going to see an agent, a vault agent injector or, sorry, a vault agent show up inside of our pod. So here's the annotation. If I make this a bit bigger, maybe to see better, you can see the agent inject is true what we saw in the slides, basically the different agent injector annotations that we expect. And here's the template where we're going to render the password, render the username and it's going to be at this particular pass app secrets that we'll see inside the container. All right, what else? Vault status from disabled to kv static injector template file and that should be good to go. We can go ahead and sync that synchronize and from here we'll wait till the API gets recreated and if we go into the logs of the API, which we'll do in just a minute and this is complete, we'll see that. Now we're using the agent injector. So let's go ahead back in here and let's make sure we are in the school app namespace. And if I run Kubectl, get all I can see. Now there are two containers running inside of my API pod. One is for the agent, the vault agent, and the other one is for my actual API application. All right, and now what we'll do is take a look at the logs for my API and you can see here credentials using Vault KV with injector and templates for vault unaware apps. You can see my username and password. This is grabbed or taken directly from vault and it's not hard coded anymore anywhere in my application code, which is amazing. So with that I can go back of course to show you that the application still runs. I'm going to re expose my ports for my application and then go back, refresh this. I can now once again add the course, I can delete a course and my application is running smoothly talking to my mongo database, no problem. So next I just want to show you what happens inside of my API pod. So if I go back here and I go into my pod, so let's clear and run kubectl exec and I'm going to exec into my pod to show you inside my pod. So inside my pod I have my main py file, of course my python file, but if I go up one directory and ls you see secrets, if I go into secrets now the path is app secrets and if you recall app secrets is through the annotations is where I'm storing my vault token and I'm storing my username and password files. So if I run cat token, this is the vault token that a vault aware app can just grab and talk to vault directly. Authenticating to vault directly. The school Mongo username is my username that gets dropped in here school app as you can see as part of the agent template. And then finally my school app Mongo password allows me to see my Mongo root pass is the password to access MongodB. And if I'm using in this case I'm using static secrets because I'm talking to the static KV secrets engine in vault. But you can always go to a dynamic secret engine and I highly recommend doing that. MongodB Vault has one for MongoDB and what will happen is every time the application needs to access Mongo, it can create on demand just in time credentials from vault to use to connect to Mongo. And there's a time to live for those credentials. Once the TTL expires, then the application needs to go and recreate a new set of credentials and in that case we no longer have any long lived static credentials in the environment. All of this I talk about in a lot of my courses in the tech and aid academy. So feel free to check out some of those courses. I have extensive coverage on Hashicorp vault. Once again, you can check out the vault 201 Kubernetes applications with vault. If that is something that that interests you, please take a look at that and that is the end of this talk. I hope you enjoyed it and I hope you enjoy the rest of the talks at Conf 42, dev stock ops, and I will see you in other videos.