Hashicorp Vault max_ttl Killed My Spring App

How to ensure that expiring Spring Cloud Vault dynamic database secrets are renewed, when reaching Hashicorp Vault’s max_ttl TL;DR Spring Cloud Vault does not renew your dynamic database secret which leaves the application in a broken state when these credentials expire. There are following options: Use a long enough and redeploy/restart the application often enough max_ttl If needed, increase the Vault system max time-to-live Restart the Spring Boot application when the database secret expires Implement database connection rotation (see ) next blog post for relational databases Disclaimer about Kotlin in code examples All code examples use (ideomatic) Kotlin instead of Java to make them more concise and easier to read (and personally I prefer Kotlin). The same approach of the code snippets would also working with Java or other JVM languages. Obviously, most people still use Java. That’s why all code examples will also be available in Java in a future update of this post. Challenges with Hashicorp Vault and Secrets Management (Mongooses cannot keep secrets - Image by Dušan Smetana from pixabay ) If you have a Spring Boot application or microservice, sooner or later (better sooner) you have to think about how to handle the secrets in your code in a secure manner. Fortunately, these days there are already really good tools available. The most famous and mature one is . As a Kotlin or Java developer building a Spring Boot application you are in a lucky situation, because it is really easy to get your Hashicorp Vault integration up and running by using . Vault’s concept of is probably THE killer feature and allows you for example to easily get unique, secure and short-lived database credentials which follow the . secrets as a service Hashicorp Vault Spring Vault dynamic secrets principle of least privilege There are quite good examples how to get started with Spring, Vault and e.g. , , and . These examples work with surprisingly little effort. But as usually the also applies here. There a quite some challenges. PostgreSQL MySQL MongoDB many other databases Pareto principle For instance, , like it is done in most examples, is easy and fast but you should never do that in production. And this is just the most obvious operational task to tackle. Here is a (not complete) list contains of things to consider: setting up Vault in dev mode securing Vault access with https/TLS setting up a certificate authority (CA) for your https connection ( , but you have to solve the bootstrapping problem) Vault itself could support here ensure that Vault’s availability is good enough to not break your system: probably you want to have a high availability (HA) setup bootstrap Vault in a secure but still maintainable way which fits your needs(aka ) unsealing who and how is the data in vault configured and maintained which Vault authentication method to use and how to bring the needed data to your applications (also here, Vault has different supporting tools like ) token wrapping There are a lot of operational things you should think about but in this blog post I want to focus on an application-level challenge with using Vault together with Spring: What happens when the database secret reaches its ? maximum time-to-live (TTL) Spring Boot and the maximum TTL of a lease in Hashicorp Vault Most people don’t think about the maximum time-to-life of this secret or expect that the database credentials are automatically rotated by Spring when needed. Let’s check the : Spring Cloud Vault documentation Spring Cloud Vault does not support getting new credentials and configuring your with them when the maximum lease time has been reached. That is, if of the Database role in Vault is set to that means that 24 hours after your application has started it can no longer authenticate with the database. DataSource max_ttl 24h This is a show stopper, because it will leave the Spring application in a broken state where no database communication is possible anymore. So, what can you do about this? I will show you 4 solutions on how to make the setup work. The first 3 are more generically applicable with some prerequisites to apply. The 4th solution is only working for a more specific but also not too uncommon setup and provides a smoother experience. I will describe its details in the next blog post. Let’s check our options. Use a long enough for the dynamic database credential max_ttl The Spring Cloud Vault documentation talks about the of the database role in Vault, which is the maximum lease time. This duration is configurable. So, if you regularly deploy or restart your application, you can just use a long enough . max_ttl max_ttl Let’s consider you have a two-week sprint and it is guaranteed that after each sprint your application is redeployed. Then you could just configure the to be longer than 2 weeks, for instance 16 days. A good example how to setup Vault to generate dynamic credentials for PostgreSQL can be found on the . You just have to adapt the value to 16 days (384 hours) : max_ttl Hashicorp Learn site about Dynamic Secrets max_ttl when creating the database role vault write database/roles/ db_name=postgresql \ creation_statements=@readonly.sql \ default_ttl=1h max_ttl=384h readonly But you cannot increase the maximum time-to-live for the secret infinitely. Hashicorp Vault has a system-wide max: The system max TTL, which is 32 days but can be changed in Vault’s configuration file. — Vault Concepts: Tokens When the Vault system maximum TTL is not enough… (Time is running out - Image by Monoar Rahman Rony from pixabay ) …it is also possible to increase it via the Vault configuration. The mentions that the parameter, which defaults to can be used for that. Vault Configuration documentation max_lease_ttl "768h" An example vault config could look like: max_lease_ttl = # (1) storage { path = } listener { = tls_disable = 1 # (2) } "3000h" "file" "/var/vault/vault-storage" "tcp" address "127.0.0.1:8200" (1) The system max TTL is set to 3000h here, the rest of the config is just for demonstration purpose (2) You should never do that in production Before you now just change the for all your Vault instances to a really big value, you should wait a minute. Hashicorp of course has a good reason to limit the maximum TTL and to not use a too big value by default: . So if you choose a too big time-to-live value, the risk of having really long living secrets and with that the risk of is increasing. That is the reason why I would not recommend changing this value if you can use one of the other options. max_lease_ttl The longer a dynamic secret lives, the less dynamic it is in the end secret sprawl Restarting the application when credentials expire If you cannot guaranty that your Spring application is redeployed or restarted often enough or you don’t want to live with the risk of long living secrets the first two options are not for you. Instead you should change the setup, so that the database secrets of the application can be rotated. The easiest way to do that is to ensure that the application is restarted when the secret expires. Depending on your setup you already have a process manager like systemd or a container-orchestration system like Kubernetes which ensures, that your application is always running. Whenever the application stops this tool will start a new instance for you. So how can the Spring application detect the secret is expired and cannot be renewed anymore? This can be done by adding an additional to the . LeaseListenener SecretLeaseContainer I expect that you’ve already setup Spring and Vault to create dynamic database secrets (how to do that, see for example or ). Then you can autowire the , the database role which is configured as the property and the to allow closing the ApplicationContext which eventually shuts down the Spring application: Managing your Database Secrets with Vault @ Spring blog An Intro to Spring Cloud Vault @ Baeldung SecretLeaseContainer spring.cloud.vault.database.role ConfigurableApplicationContext ( leaseContainer: SecretLeaseContainer, databaseRole: String, applicationContext: ConfigurableApplicationContext ) { @Configuration class VaultConfig private val @Value( ) "\${spring.cloud.vault.database.role}" private val private val In a method you can then add the additional which does the shutdown: @PostConstruct LeaseListenener { vaultCredsPath = leaseContainer.addLeaseListener { event -> (event.path == vaultCredsPath) { log.info { } (event.isLeaseExpired && event.mode == RENEW) { log.error { } applicationContext.close() } } } } @PostConstruct private fun configureShutdownWhenLeaseExpires () val "database/creds/ " $databaseRole // (1) // (2) if // (3) "Lease change for DB: ( ) : ( )" $event ${event.lease} if // (3) "Database lease expired. Shutting down." // (4) build the creds path by using the autowired databaseRole is a SAM interface, so just provide a lambda (see ) LeaseListenener Kotlin’s SAM Conversion , and are extension methods (see next code snippet) event.path event.isLeaseExpired event.mode shutdown the Spring application And here are the extension methods used above: SecretLeaseEvent.path () = source.path SecretLeaseEvent.isLeaseExpired () = SecretLeaseExpiredEvent SecretLeaseEvent.mode () = source.mode private val get private val get this is private val get If you don’t use a tool like Kubernetes or systemd you should , instead of shutting it down. restart the application The complete shutdown example can be found as a . github repository Next steps (Cliffhanger … to be continued - Image by Shri ram from pixabay ) There is also the possibility to implement the database credential renewal in the application itself. Spring itself does not provide a generic way to do this. In the next blog post I’ll show a way how to do this manually if you use Spring with HikariCP as JDBC connection pool, which works for example with PostgreSQL or MySQL databases. Originally published at https://secrets-as-a-service.com on January 28, 2020