Hashicorp Vault max_ttl Killed My Spring App

How to ensure that expiring Spring Cloud Vault dynamic database secrets are renewed, when reaching Hashicorp Vault’s max_ttl

TL;DR

Spring Cloud Vault does not renew your dynamic database secret which leaves the application in a broken state when these credentials expire.

There are following options:

1. Use a long enough 

   max_ttl

    and redeploy/restart the application often enough
2. If needed, increase the Vault system max time-to-live
3. Restart the Spring Boot application when the database secret expires
4. Implement database connection rotation (see next blog post for relational databases)

Disclaimer about Kotlin in code examples

All code examples use (ideomatic) Kotlin instead of Java to make them more concise and easier to read (and personally I prefer Kotlin). The same approach of the code snippets would also working with Java or other JVM languages.

Obviously, most people still use Java. That’s why all code examples will also be available in Java in a future update of this post.

Challenges with Hashicorp Vault and Secrets Management

(Mongooses cannot keep secrets - Image by Dušan Smetana from pixabay)

If you have a Spring Boot application or microservice, sooner or later (better sooner) you have to think about how to handle the secrets in your code in a secure manner. Fortunately, these days there are already really good secrets as a service tools available. The most famous and mature one is Hashicorp Vault. As a Kotlin or Java developer building a Spring Boot application you are in a lucky situation, because it is really easy to get your Hashicorp Vault integration up and running by using Spring Vault. Vault’s concept of dynamic secrets is probably THE killer feature and allows you for example to easily get unique, secure and short-lived database credentials which follow the principle of least privilege.

There are quite good examples how to get started with Spring, Vault and e.g. PostgreSQL, MySQL, MongoDB and many other databases. These examples work with surprisingly little effort. But as usually the Pareto principle also applies here. There a quite some challenges.

For instance, setting up Vault in dev mode, like it is done in most examples, is easy and fast but you should never do that in production. And this is just the most obvious operational task to tackle. Here is a (not complete) list contains of things to consider:

• securing Vault access with https/TLS
• setting up a certificate authority (CA) for your https connection (Vault itself could support here, but you have to solve the bootstrapping problem)
• ensure that Vault’s availability is good enough to not break your system: probably you want to have a high availability (HA) setup
• bootstrap Vault in a secure but still maintainable way which fits your needs(aka unsealing)
• who and how is the data in vault configured and maintained
• which Vault authentication method to use and how to bring the needed data to your applications (also here, Vault has different supporting tools like token wrapping)

There are a lot of operational things you should think about but in this blog post I want to focus on an application-level challenge with using Vault together with Spring: What happens when the database secret reaches its maximum time-to-live (TTL)?

Spring Boot and the maximum TTL of a lease in Hashicorp Vault

Most people don’t think about the maximum time-to-life of this secret or expect that the database credentials are automatically rotated by Spring when needed. Let’s check the Spring Cloud Vault documentation:

Spring Cloud Vault does not support getting new credentials and configuring your 

DataSource

 with them when the maximum lease time has been reached. That is, if 

max_ttl

 of the Database role in Vault is set to 

24h

 that means that 24 hours after your application has started it can no longer authenticate with the database.

This is a show stopper, because it will leave the Spring application in a broken state where no database communication is possible anymore. So, what can you do about this?

I will show you 4 solutions on how to make the setup work. The first 3 are more generically applicable with some prerequisites to apply. The 4th solution is only working for a more specific but also not too uncommon setup and provides a smoother experience. I will describe its details in the next blog post.

Let’s check our options.

Use a long enough 

max_ttl

 for the dynamic database credential

The Spring Cloud Vault documentation talks about the 

max_ttl

max_ttl

Let’s consider you have a two-week sprint and it is guaranteed that after each sprint your application is redeployed. Then you could just configure the 

max_ttl

max_ttl

vault write database/roles/readonly db_name=postgresql \
        [email protected] \
        default_ttl=1h max_ttl=384h

But you cannot increase the maximum time-to-live for the secret infinitely. Hashicorp Vault has a system-wide max:

The system max TTL, which is 32 days but can be changed in Vault’s configuration file.

— Vault Concepts: Tokens

When the Vault system maximum TTL is not enough…​

(Time is running out - Image by Monoar Rahman Rony from pixabay)

…​it is also possible to increase it via the Vault configuration. The Vault Configuration documentation mentions that the 

max_lease_ttl

"768h"

An example vault config could look like:

max_lease_ttl = "3000h" # (1)

storage "file" {
  path = "/var/vault/vault-storage"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1 # (2)
}

(1) The system max TTL is set to 3000h here, the rest of the config is just for demonstration purpose
(2) You should never do that in production

Before you now just change the 

max_lease_ttl

Restarting the application when credentials expire

If you cannot guaranty that your Spring application is redeployed or restarted often enough or you don’t want to live with the risk of long living secrets the first two options are not for you. Instead you should change the setup, so that the database secrets of the application can be rotated.

The easiest way to do that is to ensure that the application is restarted when the secret expires. Depending on your setup you already have a process manager like systemd or a container-orchestration system like Kubernetes which ensures, that your application is always running. Whenever the application stops this tool will start a new instance for you.

So how can the Spring application detect the secret is expired and cannot be renewed anymore? This can be done by adding an additional 

LeaseListenener

SecretLeaseContainer

I expect that you’ve already setup Spring and Vault to create dynamic database secrets (how to do that, see for example Managing your Database Secrets with Vault @ Spring blog or An Intro to Spring Cloud Vault @ Baeldung). Then you can autowire the 

SecretLeaseContainer

spring.cloud.vault.database.role

ConfigurableApplicationContext

@Configuration
class VaultConfig(
        private val leaseContainer: SecretLeaseContainer,
        @Value("\${spring.cloud.vault.database.role}")
        private val databaseRole: String,
        private val applicationContext: ConfigurableApplicationContext
) {

In a 

@PostConstruct

LeaseListenener

@PostConstruct
private fun configureShutdownWhenLeaseExpires() {
    val vaultCredsPath = "database/creds/$databaseRole" // (1)
    leaseContainer.addLeaseListener { event ->  // (2)
        if (event.path == vaultCredsPath) { // (3)
            log.info { "Lease change for DB: ($event) : (${event.lease})" }
            if (event.isLeaseExpired && event.mode == RENEW) { // (3)
                log.error { "Database lease expired. Shutting down." }
                applicationContext.close() // (4)
            }
        }
    }
}

1. build the creds path by using the autowired 

   databaseRole

2. LeaseListenener

    is a SAM interface, so just provide a lambda (see Kotlin’s SAM Conversion)

3. event.path

   , 

   event.isLeaseExpired

    and 

   event.mode

    are extension methods (see next code snippet)
4. shutdown the Spring application

And here are the extension methods used above:

private val SecretLeaseEvent.path get() = source.path
private val SecretLeaseEvent.isLeaseExpired get() = this is SecretLeaseExpiredEvent
private val SecretLeaseEvent.mode get() = source.mode

If you don’t use a tool like Kubernetes or systemd you should restart the application, instead of shutting it down.

The complete shutdown example can be found as a github repository.

Next steps

(Cliffhanger …​ to be continued - Image by Shri ram from pixabay)

There is also the possibility to implement the database credential renewal in the application itself. Spring itself does not provide a generic way to do this. In the next blog post I’ll show a way how to do this manually if you use Spring with HikariCP as JDBC connection pool, which works for example with PostgreSQL or MySQL databases.

Originally published at https://secrets-as-a-service.com on January 28, 2020