Welcome to the secret to handling dynamic secrets. It usually starts with, a security requirement. Your security team realizes that you've been running in cloud for quite some time, and it's no longer sustainable to copy paste a bunch of API tokens and database passwords across different applications that you've been developing. So they tell you have to migrate everything to a secrets manager. And this seems, mostly easy, I guess you have to just find every single instance of a secret and paste it into a secrets manager. But it turns out this is a little bit more complicated than someone might think, right? The minute you centralize secrets in a secrets manager, whether it be an API token or a database password, your application now has to go retrieve it over the network. And there are a number of concerns with this. Whether you're a platform engineer or developer who is responsible for managing their own cloud native application and the infrastructure underneath it. There are a number of concerns you might have with an application reaching out to a secrets manager, whether that be network connectivity or just generally the availability of your password and API token. And it turns out it gets a little bit more complicated than that, right? What happens when you have to rotate the secret, right? Imagine you have a security requirement number 208 or something that means that you have to rotate that secret every 30 days. now you have to coordinate with your application. Your application has to reload, reconnect to the secrets manager, and connect to the second version of the secret. Ultimately, after some buffer period of maybe four or five days, you'll delete the old version of the secret just to make sure your application is working. This works Okay, for maybe one application, if you're just managing a single application that's connecting to the secrets manager, but what happens when you grow your environment and now you have multiple applications waiting to connect to a secrets manager to get an API token and a database password? it's a little bit challenging to figure out which applications need to retrieve which password, which applications need to access the API token. And if you ever have to rotate this password or this API token again, you now have to make sure that every single application that is potentially using this shared API token and password have to reload. Now there are a couple solutions around this, right? And the first is, trying to automate some of this. I spend a lot of my time automating and trying to figure out better workflows for building security into applications and infrastructure. And it turns out that there's one really big secret to doing dynamic secrets. And when I talk about dynamic secrets, the ability for an application to handle a new secret that it has to use. The secret is actually investing in some developer enablement. There is no magical tool, and unfortunately, no magical workflow. That will automatically do this for you. Unfortunately, this requires an investment into enabling developers, providing better documentation. It also means that you have to understand the patterns that are involved in order for an application to support and a constantly rotating secret. The reality is, in a cloud native application, you have to be responsible for handling dynamic configuration. You expect any dependencies to change, right? And why shouldn't we depend, why shouldn't we make sure that we are able to depend on an ever changing secret? The answer is, we need to have a little bit of a paradigm shift in how we develop our applications to handle secrets. Traditionally, we are very used to having one secret that we never touch and hopefully will never have to change. But in a cloud native environment, that's not possible to expect that, right? From a security posture perspective, you want to make sure that you are rotating your secrets on enough of a frequency so that you do not have to worry about someone using those credentials or an unauthorized service using those credentials. So investing in developer enablement is a big portion why you have to have some kind of secret and handling dynamic secrets. As I mentioned before, there are some categories of secrets though, right? We traditionally think of a static secret where we manually copy and paste this somewhere, probably into a secrets manager, and then we manually delete the old secret. This is all done manually. And this works perfectly fine. You treat a secret like a key and a value, and you inject it into your application, right? Now, more and more platforms, especially in the cloud native environment, are offering rotated secrets. This means that automation retrieves and stores the new secret. And you manually decide to delete the old secret, right? So this is more common if maybe you're seeing a managed database or some other managed service that's offering an API token or a password. It will help rotate it for you on your behalf. You'll just have to make sure in your application that you're able to consume the new secret. Finally, there's the dynamic secret. The dynamic secret means automation both generates and revokes the old secret. I'm going to be showing HashiCorp Vault, which is a secrets manager. In as a demonstration of some of these patterns, but dynamic secrets do exist quite a bit across large cloud nativist states in which you've built significant automation to handle this, right? So if you have a dynamic secret, automation will generate the new secret for you, and it will revoke the old secret on some kind of cadence. These are the three categories. It goes from more manual to, of course, more automated. It doesn't mean that every secret is going to fit into a dynamic secret, right? You can and you will have secrets, even in a cloud native environment. That are static. And that's because there are some systems that will always have an API token that you need to manually and go and generate. Unfortunately, this is part of the systems that we work in. It's part of integrating systems in a large organization as well. So you will always have some kind of static secret, but more and more, especially as you have, services that are enabling the rotation of API tokens using an API, you will get something like a dynamic secret. Today, we are going to learn more about dynamic secrets, but the patterns that you learn here do apply to static and rotated secrets. That's because we are trying to describe patterns for the most extreme scenario in which most secrets are rotated frequently and you expect them to change, right? So if we're talking about patterns for a dynamic secret, the same rules will apply to static and rotated secrets, mostly because this is about how an application reacts to a secret dependency change. So what does it mean to handle a dynamic secret? There are three parts to the workflow. An application must get the secret. This is actually divided into two substeps. The first is authenticating to the secrets manager and then, using the secret, right? So once you authenticate to the secrets manager, you can get the secret and you can use it in the application. The second part of the workflow is actually about changing the secret. The secrets manager or maybe yourself will change the secret. You'll rotate or expire the secret. In the secrets manager, the application has no awareness that you've done this unless you tell the application that you have actually changed the secret, right? And telling the application that you've changed the secret means you either reload the application or the application detects the change the secret and reloads code. This is very important to know, especially if you have dynamic secrets. These three parts of this workflow are required for any kind of workflow. rotation or any kind of changing secret that you have in your application. It's not much different than let's say if you had dynamic configuration, right? If you have an application that is retrieving dynamic configuration from a configuration server, this workflow looks pretty similar, right? Authenticate to a configuration server, retrieve the configuration properties. If those configuration properties change, then you must reload, right? It's a common enough pattern for any cloud native application that's built to be cloud native, but we just typically don't treat secrets like this because, secrets don't change as often, or traditionally we haven't changed them that often. So the pattern still applies even if you're doing a static secret or you're doing a rotated secret. It doesn't have to be dynamic. Patterns for dynamic secret can be broken down into two, two separate patterns. The first is that the application is agnostic of the secrets manager. What does this mean? the application doesn't request secrets from the secrets manager. It doesn't connect to the secrets manager at all. Actually, it refers to a separate process. The separate process authenticates to the secrets manager and then injects it. We'll go more into this. a little bit later. Then there's the application aware approach. The application itself is aware that there's a secrets manager and it needs to make an API call to the secrets manager to get the secret. So this means that you use some kind of API or SDK request. Both patterns are valid. I do hear a whole number of debates on whether or not one is better than the other, and I can certainly tell you there are benefits as well as disadvantages to each, and we'll cover some of that in more detail once we describe some of the technicalities and the differences between the two. So let's start with application agnostic. As I mentioned before, application agnostic means that there is a separate process that is handling the authentication and the injection of the secret. Usually this injection is in the form of environment variables or files. You get to choose. Most applications are more familiar with environment variables, and so the application will read the secrets from the environment variable. Overall, it's a pretty straightforward process, right? You've got a separate controller or separate process that is going to handle interfacing with the secrets manager, and the application doesn't have to change. Nothing has to happen. It just continues to read from environment variables as it wants to. This works pretty decently for a number of applications across many different frameworks. It works really well for older applications that you might replatform into a container, but you don't necessarily want to refactor the code, right? So when the password or the API token changes, the separate process, handles the retrieval of the new secret. The separate process is constantly monitoring the secrets manager. It could be on a schedule, or it could be some kind of event subscription. And it's going to retrieve the new secret, re inject it into the environment variable or file, and then the application picks it up from the environment variable or file. Application frameworks like Spring Boot or NET, they do have some kind of live reload configuration properties from a file. So that means that you can reload the application once it detects a difference in the configuration properties or file, which makes it more straightforward. If you have a programming language or framework that you're not sure if it supports live reload, you may have to write it yourself. But effectively, you would be looking for a file difference or some change the environment property, and then you would be reloading. Now, there are other ways that you can think about doing this. And it's not just, going to involve some kind of separate process. if you are on Kubernetes, for example, We're going to do this. if you're on Kubernetes, for example, you may have a separate process that actually reloads the application on your behalf, right? The application itself doesn't have to do a live reload, doesn't have to detect a difference in the file. Instead, you can have some kind of separate process reload the application for you, right? It could, either kill the process that the application process ID that the application is running on, or it can send a signal to the application. I need you to refresh something. a good example of this that I'll mention is that if you are using the spring boot actuator, in the spring boot framework for your development teams, the actuator has a refresh endpoint. You can make an API call to refresh. values in the application, right? Other frameworks do have some kind of refresh capability as well that you can maybe build as an API endpoint and then expose that for your separate process to handle the reload for you. So you don't have to necessarily support that in the application itself. You can do that outside the application. So once it reloads, then it will get the new secret, and that's pretty much all it needs to do. This is a pattern that's pretty common for Kubernetes. If you are running on Kubernetes, you often will find that there's a number of Kubernetes operators out there. That will load in external secrets from something. it will load in an external secret from a secrets manager. It will handle the authentication on your behalf. It will retrieve the secrets that you need, and then it will inject the secret as a Kubernetes secret. So the Kubernetes secret can be added as a file volume or it could, be added as an environment variable. In newer versions of Kubernetes, it is suggested to add them as volume amounts. but not every application supports, reading from files and there are some things that you do have to refactor on the application side in order to potentially read from a file. Most applications standardize, at least in the past, in the past on environment variables and expect to do environment variable injection as a default. So there's, there are two choices, but both are supported. So you can do a Kubernetes secret as a file, or Kubernetes secret as environment variable. I'll show the environment variable example. So the important thing is that this application is a Spring Boot application. it's not as relevant, but it's important to recognize that it is a Java application, and it does expect to have environment variables like a database username, a database password, a static payments processor username, and a static payments processor password. All of these in my Kubernetes manifest for my deployment are injected by environment variable. I read the value from a Kubernetes secret called payments app database with the key username, with the password, and everything else. So there's two separate secrets that I'm referring to. One is for the database and one is for the processor. And so these are things that maybe I keep default in my application. I expect to read them as environment variables anyway. And so I don't really have to do any significant refactor in my application. One important thing to know is that the database password here is being stored in Vault, and being generated by Vault dynamically, while the processor password is going to be static, so it's stored as a key value pair. I'll show how both get processed and injected into the Kubernetes secret, but there will be two. Important thing to also note is that again, this is a Spring Boot application. I'm reading from environment variables. I could very well render this asset of application properties as a config map or something else, but I'm just injecting them as environment variables. The next thing that I'm going to do is I'll just check that my operator is up and running. the operator that I'm using today is the Vault Secrets operator. The Vault Secrets operator works with HashiCorp Vault specifically, although there are many other operators that you'll find that inject secrets from other secrets managers as well. But the operator pretty much does the function of authenticating to Vault on my behalf. which is why I have a custom resource called vault auth. It uses a Kubernetes service account. And then I can define exactly what secrets I want to retrieve from vault. This is pretty, standard of a pattern in terms of custom resources across Secrets Manager. So you'll define what kind of secrets you want. If it's a static secret, this is a key and a value. and this key value, is a password and username for my payments processor that I've stored in my Secrets Manager. So I'm able to retrieve it as I need to. And I will define my secrets destination as payments app processor and refresh after 60 seconds. This is a pretty frequent refresh, you might not need 60 seconds. You could be 30 days, you could be 15 days, right? I don't suggest you do 60 seconds unless you have a secret that always changes. And finally we have a rollout restart target. there are Kubernetes operators out there that do support some kind of rollout restart. They leverage the Kubernetes rollout capability that you can target a deployment, a daemon set, or a stateful set, and it will restart, the application on your behalf. This, Mivalt secrets operator does support that. So I'm targeting my payments app deployment. So anytime the secret changes, it will roll out and restart the application for me. And the dynamic secret is, very similar. The diff difference is that I am getting a username and password dynamically anytime I request it from Vault. So it's going to be a unique database, username and password each time. So I'm going to deploy the custom resources related to the secrets so that I can hopefully see that a secret has been created, with those custom resources, the. The operator is going to create two sets of secrets. One is for a processor and that has the username and password. The other is the database. So we'll check out the processor first. You'll notice there's a username and password field. We also have the database as well. One thing to know is that if you need a different format, you don't want username password like this. You want them in a file somewhere. There are some operators that allow you to have a custom resource definition that sort of templatizes the file type you want. I'll apply the deployment, which we're going to call payments application. And it's going to start up with an initial database username and password. In the logs, I do print them out. Do not print your username and password in plain text in logs, but I'm doing this for the sake of example. So when it first starts up, it's going to have 971, a suffix of 971 for its username, and a password prefixed with 084. And we're going to wait for a little bit. this will expire after three minutes, not every database password you'll want to expire after three minutes. most of the time you'll want to set it on a much shorter cadence. I'm doing this for the sake of example. so the way that Vault works is that it has dynamic credentials. These credentials have what is called a time to live. Vault will lease a certain set of credentials for a certain amount of time. And then after that certain amount of time, it will revoke and expire the credentials for you. It is something that you have to monitor for. You do not get any real awareness of whether or not the secret will expire unless you actively check, how long it's been, how long that secret has been around. So the lifetime of the secret is controlled by Vault. you can check out the logs for the operator. You'll notice that there was a rollout restart failure when it initially started up, but there's no indication yet that it's going to retrieve a new secret. So we'll just keep waiting until the secret gets rotated. It's been about a minute and a half and so it's your time out. Just about now. if you are not familiar with dynamic credentials and this is the first time you're working with a lease, I recommend that you wait for the lease to be two thirds of the way through its lifetime. after two thirds of its lifetime, then rotate. So it gives you a little bit of time, to expire the old credentials and make sure that you've actually your application works, but you'll notice that the new app, the application has restarted and there's a new set of usernames and passwords for the database. So this one ends in 099 and there's a password of Q9S. So that is how you can do an application agnostic approach. there are some benefits to the application agnostic approach. Of course, you don't have to refactor your application, right? My application could still happily use, environment variables from Kubernetes secrets, and it was none the wiser, right? Everything else was handled by the operator. The operator reauthenticated and helped me roll out and restart pods on my behalf, and it issued a new username and password for a database on my behalf as well. This is pretty nice. It means I don't have to go looking for where these database usernames and passwords are being referenced. Instead, what I have to do is just Make sure that it connects to the secrets manager. it makes things a lot easier from a development perspective. Now, not every development team wants to do the application agnostic approach. and there are a couple of reasons for why they would not necessarily want to do it. I encounter a number of developer communities who prefer to do everything in the code. And that's because their application is particularly performance sensitive. There are some significant performance requirements. That means that they do not accept any latency, even though Kubernetes rollout restart is a rolling update. meaning it's not going to roll everything, roll every pot at once. It is going to do it gradually. it is not sufficient from a performance standpoint for the application. So some development communities prefer to build in the connection to the secrets manager within the application itself. Other times it's partially because they might be developing a developer community that is being supported. They might be developing a cloud native application, but it doesn't necessarily run in containers. and so the result is that I have no choice but to interface directly with a secrets manager. they don't have an external automation process or external process that can handle the authentication and injection, right? Instead, the application itself has to do everything on its own. So what is the application aware pattern? The application aware pattern is a little bit different. And then the application now has to be aware that a new secret, it needs to retrieve a new secret. This is a bit nuanced, right? It's not so easy to tell an application or for an application to understand, Oh, this, the old credentials are going to expire, now I need new ones. it's not quite intuitive, right? So there are a couple patterns to understand. The first is that in your code, your application needs to authenticate to a secrets manager. Then it needs to track expiration. There are two ways in which their code could track expiration for a secret, right? So imagine you have database password version one. you can set a timer for database version one, database password version one, and say, okay, I know that it will be done. It will be gone in three minutes. So at two minutes, I am going to get a new set of secrets, right? or you can subscribe to events. So this could be a queue that someone has put together and they're saying, Oh, I'm going to make sure that I have, some kind of event that's getting published that gets sent to my application. My application will subscribe to that event and recognize that this new, this new set of secrets is going to get, is going to need to be created, right? because there's some event that's saying it's going to expire. So there are a couple important facts about this that you have to keep in mind. The first is that you need to gracefully shut down any connections. If you're using an API token or a database username and password to connect to a database or any kind of third party service. You need to gracefully disconnect everything. And this is something I have to emphasize with developers, especially as I work with them, then even working with platform teams who have to enable developers, I'm often talking to them and saying, are, do you have any kind of graceful shutdown or some kind of connection handling so that you can disconnect everything that's using the old password? And they're like, I didn't realize that. I thought I would just shut it all down. it turns out you do want to make sure you're gracefully shutting down those connections. And then you can inject new credentials for new connections. And then if, let's say the secrets manager goes down, you do have to handle retries within the application as well. So there are a number of patterns you have to consider as you do a code refactor. So there's a lot more, logic that you have to include in your application if you're doing the application aware approach. Okay, so we'll take a look at what this looks like, from an ideal perspective. Spring Boot is very nice because it offers two, I guess one, one library that allows you to inject the credentials from Vault, into your configuration, and then it does offer an SDK for you to interface with Vault. So it's very convenient. the first, library that you'll probably work with is the Spring Vault library. The Spring Vault library is the base SDK or base API call to Vault itself. And then there's Spring Cloud Vault. Spring Cloud Vault uses Spring Vault and Spring Cloud Vault injects configuration properties on your behalf. So it will retrieve information from Vault and then inject it into configuration properties for you. And they're both really fantastic examples of good patterns for doing an application aware approach to connecting to secrets managers. And so that's why we're using them today. But you'll need to add these dependencies into your application. Now, if you're using something like Go, NET, any of the other language frameworks, you may not find something that exists, right? Some libraries do handle retries, some libraries don't. do your research, double check which libraries are able to handle retries on your behalf, which ones do injection for you on start up. and make sure that you are verifying whether or not those behaviors are something you're going to need to write or your development team needs to write in your application or otherwise. so there's quite a bit of research that you'll need to do in order for this to happen. I'm including the Spring Cloud Vault, config, library, which injects configuration properties, into my application. And I'm going to be injecting the static secret as well as a database secret. In my application properties, the biggest thing that you have to define is authentication to vault, which are lines five and six, which is the secrets manager. There are other ways in which you can define. Some kind of authentication if you need to your secrets manager in this case This is a configuration property that you'll set for the library itself But if you're using other libraries for other secrets managers, you may find some differences the other thing that I do is I have to set a configuration server to point to my secrets manager. Again, this depends on the library that you're using. I'll enable the database secrets that I'm going to retrieve as well as key value secrets that I'm going to retrieve. Again, these are more specific to Vault itself. So check your library for your secrets manager to determine which attributes you need to enable. I've already stored the secrets in Vault. So we'll pull up the secrets. I'm sorry. They're not at that secrets. They're not at that path, but they are going to be in the secret path that is going to match the application name. and I've. I already stored them as formatted as spring application properties, right? So custom. staticsecret. username are custom properties that I'm defining, and they are in the form of spring properties. You can choose not to do this. I'm just doing this for convenience, especially as I'm using the Spring Boot application framework. So once I set all of that up, I can now, refactor my code. if you, again, if you have other frameworks that you're using, they don't have an SDK like this or they don't have a library like this for your use, you will have to build some of this yourself. You will have to go and authenticate to Vault or a secrets manager, retrieve the secret, and inject it. so once you, once you get that process done, the second thing you need to do is, of course, handle a refresh. and as I mentioned before, a second part of the workflow is, detecting when things need to be refreshed. the first is that if you have a static secret, you might choose to do a scheduled refresh. And a scheduled refresh works perfectly fine, especially if you have a password that's not going to change that often. Now, if you have a password that you're planning to change every 10 minutes, This might not be ideal, but for a more simplistic implementation, you can just schedule delay and then refresh the application. In this case, I'm using what is called a spring, spring applications context refresh. And what this will do is refresh the application context for me. I do have to tell it what objects to refresh and. Exactly what needs to be recreated, but it will issue what is called a context refresh for me. Again, you can change this to do some kind of event subscription if you want to. for example, you can detect whether or not a secret has changed in the Secrets Manager, publish that event, and then your application can subscribe to it, but most secrets just can handle a scheduled refresh. They don't need to have a real time event based refresh. The other thing that you might want to consider, especially for dynamic secrets, if you know that There is a secret that's going to expire within a certain period of time. Let's say three minutes. you can of course create some kind of event system for you to find some kind of lease expired event. so in this case spring Vault has a secret lease expired event as a lease listener that I can, I can subscribe to. And then if I detect a secret lease expired event, then I can refresh the application. So this is the second way to do it. Again, if you're using a static secret or a rotated secret that has some kind of frequency, like lower frequency in rotation, you could probably just do a scheduled task to reload. The application with that secret, or if you have a requirement for a more real time approach, you can do some kind of event subscription, and that's what this is showing. Either way, both patterns do work. It's just a matter of making sure that you are able to implement it properly in your code. So after you've set up some kind of refresh mechanism, whether it be on a schedule or you're subscribed to some kind of event, the next thing you do need to do is identify how to reload. all of your objects, right? and that's where this is probably the most important part of the code refactor that most development teams and platform teams that I work with tend to forget. I'm going to enable scheduling, since I am scheduling a task to reload a static credential. the most important part that I would say developers and platform engineers when we're talking about dynamic secrets. The ones that kind of forget is actually refreshing the credentials into objects that are using them. What I mean is that it's not enough to, for the application to refresh and reload those usernames and passwords into some kind of property list. You need to re inject those properties back into any objects that use them. Good example of this is like data source, right? in a Java data source. It needs to connect to the database and it needs a username and a password. I have to recreate the entire data source with the new username and new password. This is incredibly important because if I don't, then my application will continue to use the old username and old password to continue connecting to the database since. It's not been refreshed. The entire object has not been refreshed. So I use what is called a Spring, Spring Boot's refresh scope annotation to basically let Spring Boot know you need to refresh this entire object and you need to make sure that you're using the latest username and the latest password from properties and rebuild the entire data source on my behalf. The underlying implementation does a graceful shutdown of connections. so I don't have to necessarily implement that myself. But if, again, you have a framework that is not Spring Boot, and you have a framework that you have to implement this all yourself, you do have to have some kind of graceful disconnection capability built into the code base. and you can do that by abstracting that into a library, some library that you've written yourself. but do keep in mind that it's not just a combination of detecting the event of needing to refresh or change the secret. The secret itself also needs to get re injected into any, client connection or data connection that's using it. So those are two critical, pieces of the workflow that I often see are forgotten, especially when we're talking about, refreshing applications to account for dynamic secrets. All right, so I'm going to run the application, just to show exactly what this looks like. unlike the, Kubernetes example that I showed, this will refresh within the application, meaning the application does not restart. It just simply reloads the, credentials. So the first database credential has a username suffix of 8443 and a password prefix of F A. And I have an expiration of two minutes for these database credentials. So after about two minutes, this application is going to rebuild the data source with a new set of credentials. What's important to recognize about a number of the, these applications that do have, are aware of the secrets manager. You do have to handle any retries. if let's say the secrets manager is not on, not online, and you can't connect to it, right? You have to make sure that you can cache the secrets, existing secrets. If the secrets manager is offline, it's not going to rotate the secret for you, so things are fine for the most part. the biggest problem is actually whether or not the application itself can cache these secrets, especially if you're doing some kind of scheduled refresh, right? you don't want to override the previous secret. Ideally, you want to keep the old secret cached until you can reconnect back to the secrets manager. The other important thing to know is that if you are doing this within a framework that doesn't handle graceful shutdown, you will have to handle the graceful shutdown. Make sure that you're clear on what that sequence looks like and make sure you're clear on what handling you want in your application. it's not necessarily just databases, right? So we've now got the database credentials refreshed. I'm going to create, make an API call to the application. in the case of Spring, you have to make an API call, in order for it to reconnect to the database. So it rebuilt the secret with a new set of secrets, right? Different, suffix you'll notice for the username and different password. it's got a new username and password and now it's restarting a Hikari pool or a database connection pool. and so all of these are quite important steps in the process. Especially if you have application aware approach. All right. So I didn't mention before there are benefits and disadvantages to each with application agnostic. You're able to support many languages and frameworks and you're maximizing platform automation. So I see this pattern quite a lot in the kubernetes ecosystem. most of the time when you have heterogeneous programming language types, different types of containers running in a kubernetes cluster. most languages. Codebases don't really want the work of refactoring, their code in order to support a live reload. and for the most part, the availability of the application will depend on the number of instances you have. Most of the time you're running multiple replicas of the application in Kubernetes or some kind of cloud native environment. So a rollout, that has like a rolling update capability is not going to disrupt the application significantly, right? So the good news about application agnostic approach is that you can maximize your platform automation. This makes it a little bit easier for it to be maintained by a platform engineering team. However, there are reasons that I've seen for people to do application aware. There are organizations that I've worked with who Standardized on very few languages and frameworks, right? They're all committed to spring boot or they're all committed to dot net, and they would rather invest in code libraries or their own libraries to handle all the code because they prefer to control the performance as well as the behavior of their applications. In response to changes to dynamic secrets, right? And so the availability will depend on the code base. You're only as good as the code that you've written from a performance standpoint. So there's a lot of testing involved. There are a lot of patterns going back and forth and trying to evaluate. A lot of stress testing involved when there's a third party or a new library that's being introduced. but if there are a few languages, only a few languages that are approved, and most applications are being developed in a single language, then I do see organizations who go with this application aware approach, and they would rather refactor the application. The, downside is that it doesn't maximize code refactor. So you do have to take some time. So refactor your code base to handle and reinject the secrets. And this is something that is not going to be. Let's say, abstracted away by a platform team building like us the equivalent of, let's say, like Spring Cloud Vault for their chosen framework. it is something that has to be done within the application itself. And I think that's the challenge that I typically encounter with development communities and organizations who are trying to do this. they're saying, Oh, I, I know I need to reload this object with a new username and password, but I've done it in a nonstandard way and we've had this code for four years. It's not like I'm going to change it. So there are some nuances there and that most of the time platform teams are trying to navigate and understand. it does minimize platform automation, so there is a good benefit in some ways to this in that if your platform is particularly resource constrained, you don't want to be, adding more separate processes to your cluster to your container orchestrator or something, this application aware approach is a way to alleviate some of the burden and put it into code, but there's two approaches to it. The cloud native approach, of course, is to think about doing this from an application agnostic perspective, meaning that the application itself should not have any significant or understanding of the interface to the secrets manager. But there are reasons for folks who do want to do application aware, even if they do have a cloud native application. Before you choose either application agnostic or application aware, though, if you're a platform engineer or you're a development team that's the center of excellence for things, or a developer community who's, trying to standardize and trying to figure out what's best, Do assess your applications. I think this is the biggest mistake that I've seen. Most folks don't really know what applications are running. Document what your run times are, where are the applications running, how they're running, are they in containers, are they not in containers. There are a lot of systems that don't have containerized applications and it's a mix of the two, ideally in cloud native, of course, we take we tend to, we tend to think of cloud native as containers only, but that's not always the case. you is possible on occasion to have an application that follows some cloud need of principles, but not necessarily running in a container. And so do assess the runtime. Do assess the frameworks that you are using. And if you're using spring boot, then you have a lot of libraries available to you that handle injection as well as authentication to secrets managers. There are many of them, and there are very, they are very convenient to use, but make sure you understand their behaviors if you have, other frameworks that don't necessarily support a retry, doesn't support any kind of interfacing with your secrets manager. You may have to write some additional code on your own. So you do have to assess what frameworks exist. And then availability requirements. Some applications do not, do not perform well, with an application aware approach. they are, and they perform better with an application agnostic approach. So they are horizontally scalable. They are able to be reloaded and rolled out with a rolling update. Oops. With a rolling update, while others do not, right? So it's dependent on your availability requirements for your application. And then finally, secrets. Secrets, there are many types. So you know, we talked about database and API credentials, but secrets injected into applications can range from certificates. to some other kind of identification that allows access, right? if you have, let's say, AWS and you're on a cloud service provider, they may have their own unique, authentication and instance information, metadata information and things like that for you to authenticate. And you also want to document the rotation frequency of secrets that are being used by your application. I would say most of the time, I don't get a solid answer on what the rotation frequency of application secrets are. Most of the time I get, an explanation of, we only rotate it on demand, which is never. We've kept this secret around for three years and we've never touched it. and that's perfectly fine. It's just good to document, which secrets you may expect to rotate more frequently and which types. Database credentials, for example, you may expect to be, asked to rotate them every 30 days from a security perspective. certificates, you could have a lifetime of 10 years, but certificates themselves also expire. and certificates, may have a rotation frequency of 5 years or a year, depending on the certificates. documenting how these secrets are working within the application is very important. Doing this discovery phase will help alleviate a lot of the questions and inform whether or not you should be thinking about an application agnostic approach or an application aware approach, when you think about having, dynamic secrets or any kind of interfacing with secrets in your application. And finally, developer enablement. After all, I said that was the main secret. all of the patterns that I taught here in this session, you can take back, right? Take back and build documentation around them. These are the most common patterns that I've seen. For developers to inject secrets into their application and handle more dynamic or dynamic secrets that are changing more frequently, right? so document these common patterns, organize the workflows and articulate and communicate them to your development communities, learn them for yourself as well. and provide lots of code examples. Most developer communities within an organization or who are trying to build their cloud native application, they do want a lot of code examples, right? Office hours are pretty critical, and I would say do one or two live refactors with certain development teams. This has been really helpful. in my experience, because you get a sense of what the development team patterns are, and you also get to identify any pitfalls that you might not have thought about. a good example of this, as I was working with a develop, a development team, and they had some specific patterns in Spring Boot, and they didn't use beans. They didn't use the annotation for Bean, which meant that refresh didn't quite work right for them. it was good to understand what they were looking for. other teams that I had worked with, they are using, let's say, GraalVM or Native Image, and they couldn't use Reflection, Java Reflection, right? we had to write some code from scratch, to help, detect whether or not those credentials were changed and then refresh, refresh these objects by local proxy. like I said, this sounds maybe complicated or if you're not familiar with Spring. doesn't make any sense. But the reality is there are only patterns that you're going to find during live refactor. and it's good to learn those new patterns, identify what the development communities are using, what they're not using. and it's helpful to build videos around certain patterns, as well. I think a lot of the, platform teams that I've worked with, they're like, Oh, can we just make a video out of this? Like we'll just cut and, cut away some of the pieces. But the most important thing for us is we just want to make sure that we distribute this to our developer community so they can see this live refactor. And finally, from a development perspective, you may have to build new interfaces in order to support this paradigm of dynamic secrets. sometimes you don't have frameworks that support a retry, a proper retry. Other times You know, there are new frameworks out there that are coming out. and then you can't, you just don't have the ability at that point to support retrieval from secrets managers, right? So you have to write your own API and SDK for it. so build the interfaces if you need it and certainly assess any exceptions. there are a number of communities that I've worked with who went down the application aware path, only to find that it doesn't really work for their application. maybe a small portion, like 20 percent of their applications. And so they'll end up doing the application agnostic approach for that 20%. This happens for, maybe not necessarily cloud native applications, but applications that Cloud Native, the Cloud Native environment runs or accesses, right? These are services that are not Cloud Native themselves, but are heavily dependent on by a Cloud Native service. these could be date, older databases, or they could also be, older APIs. So do assess any exceptions that you do have. You will have a combination of not application aware, but also application agnostic approaches, right? Because not every framework will support it and not every application will be able to interface directly with the secrets manager. And similarly, in the application agnostic approach, you might not find the capability exists in your runtime, right? Depending on the container orchestrator you're using or just workload orchestration in general. Finally, that's the end of The Secret. it's a lot. It's a pretty big secret, actually. but I'm trying not to keep it a secret any longer. So if you do have any questions, you're welcome to reach out to me. All this material is available on my webpage. So feel free to review it and distribute it to your own development communities. If you're a developer who wants to handle dynamic sequencing your application, hopefully you have a sense of the patterns that you need to implement. If you're a security team or a platform engineering team that needs to identify and assess and educate your development community on how to use a centralized secrets manager. hopefully this provides some valuable information on how to improve the overall adoption of your secrets manager and make it a little bit easier for your developers. Thank you.