paint-brush
Heavy Rotation of Relational Hashicorp Vault Database Secrets in Springby@usr42
288 reads

Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring

by Balthasar BiedermannMarch 3rd, 2020
Read on Terminal Reader
Read this story w/o Javascript
tldt arrow

Too Long; Didn't Read

This is the second episode in a series of blog posts about how to handle the expiration of Hashicorp Vault generated dynamic database credentials in a Spring application. Spring leaves your application without a database connection when these credentials expire. This time I would like to show you how to renew the database credentials at runtime for relational databases if you use HikariCP. The costs for the approach are: more implementation effort (only relational databases supported) more effort (stricter prerequisites) and more work (optional)

Company Mentioned

Mention Thumbnail
featured image - Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring
Balthasar Biedermann HackerNoon profile picture

Rotate Expiring Spring Cloud Vault Database Credentials Without Downtime

TL;DR

The first episode of this series of blog posts can be found here: Hashicorp Vault max_ttl Killed My Spring App

It is possible to rotate the Spring Cloud Vault database credentials at runtime for relational databases if you use HikariCP. To do so add a 

LeaseListenener
 via 
addLeaseListener()
 which

  • calls 
    requestRotatingSecret()
     on the 
    SecretLeaseContainer
     when the dynamic database lease expires
  • reacts on a 
    SecretLeaseCreatedEvent
     with mode 
    ROTATE
    ) by:
  • setting username and password on the 
    HikariConfigMXBean
     of the 
    HikariDataSource
  • call softEvictConnections() on the 
    HikariPoolMXBean
     to use the new credentials

New Spring Cloud Vault database secrets without downtime

This is the second episode in a series of blog post about how to handle the expiration of Hashicorp Vault generated dynamic database credentials in a Spring application. Spring leaves your application without a database connection when these credentials expire. For more context and some general solutions please check the first post.

This time I would like to show you how to renew the database credentials at runtime. So, this time you neither need to regularly restart or redeploy your application nor to use a (probably too) long maximum time-to-live for the credentials nor do you have to programmatically restart the application, which could potentially result in downtime or not met SLAs.

As we all know there ain’t no such thing as a free lunch. The costs for the approach I am presenting you this time are:

  1. more implementation effort
  2. stricter prerequisites (only relational databases supported)

The first bullet point is addressed because this post should help you with the implementation. This leaves us with the…​

Prerequisites

This approach is only applicable for Spring applications which use HikariCP.

Luckily the usual way to store and retrieve data in a relational database with Spring Boot is to use Spring Data JPA. In Spring Boot 2, Hikari is the default DataSource implementation, which makes it typical setup for using relational databases.

Show me the code

To fulfill the prerequisites, it is enough to depend on the Spring Boot JPA Starter:

plugins {
    id("org.springframework.boot") version "2.2.4.RELEASE" // <1>
    id("io.spring.dependency-management") version "1.0.9.RELEASE" // <2>
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-data-jpa") // <3>
    runtimeOnly("org.postgresql:postgresql") // <4>
}

<1> You don’t have to use the Spring Boot Gradle plugin, but it makes your live easier
<2> The Spring dependency-management plugin together with Spring Boot Gradle plugin ensures that all Spring related dependencies have the version being compatible with the Spring Boot version
<3> By adding the 

spring-boot-starter-data-jpa
 dependency together with Spring Boot 2.x you automatically get HikariCP
<4> In my example I use PostgreSQL but also most other relational databases, like MySQL, would work

These few lines are basically enough to meet the requirements of using HikariCP, in this case with PostgreSQL.

Rotating the expiring database credentials at runtime

(Rotation at runtime - Image by Peter H from pixabay)

To rotate the database credentials, which are dynamic secret from Hashicorp Vaults point of view, we have to do following steps:

  1. Detect when the database credentials are expiring
  2. Get new dynamic database credentials from Hashicorp Vault
  3. Refresh the database connections to use the new credentials

Detect when the database credentials are expiring

To detect when the database credentials are expiring we can use the same approach like we did to restart the application when credentials expire in the first blog post. Let’s again autowire the 

SecretLeaseContainer
 and the database role which is configured as the property
spring.cloud.vault.database.role
 to the 
VaultConfig
 configuration class:

@Configuration
class VaultConfig(
        private val leaseContainer: SecretLeaseContainer,
        @Value("\${spring.cloud.vault.database.role}")
        private val databaseRole: String
) {

As before in a 

@PostConstruct
 method you can then add the additional
LeaseListenener
 which does the lease rotation:

@PostConstruct
private fun postConstruct() {
    val vaultCredsPath = "database/creds/$databaseRole"
    leaseContainer.addLeaseListener { event ->
        if (event.path == vaultCredsPath) {
            log.info { "Lease change for DB: ($event) : (${event.lease})" }
            if (event.isLeaseExpired && event.mode == RENEW) {
                // TODO Rotate the credentials here <1>
            }
        }
    }
}

<1> When this code path is reached, the database secret expired

Next step is to…​

Get new dynamic database credentials from Hashicorp Vault

(The credentials should be renewed - Image by pasja1000 from pixabay)

When the lease for the database credentials expire we have to request a new secret.

if (event.isLeaseExpired && event.mode == RENEW) {
    log.info { "Replace RENEW for expired credential with ROTATE" }
    leaseContainer.requestRotatingSecret(vaultCredsPath) // <1>
}

<1> Tells Spring Vault to request a new rotating database secret

The returned value of 

requestRotatingSecret()
 is of type 
RequestedSecret
:

Represents a requested secret from a specific Vault path associated with a lease RequestedSecret.Mode.

RequestedSecret can be renewing or rotating.

— Spring Vault Javadoc

As mentioned in the Javadoc, the 

RequestedSecret
 contains the path and the mode of the secret, but it does not contain the secret itself. So how do we get the requested credentials?

We have just requested a new rotating database secret within our own 

LeaseListener
. This listener receives 
SecretLeaseEvent
s which are also created, when a new rotating secret is received. This is exactly what we need! So, let’s also react on this kind of event.

if (event.isLeaseExpired && event.mode == RENEW) {
    log.info { "Replace RENEW for expired credential with ROTATE" }
    leaseContainer.requestRotatingSecret(vaultCredsPath) // <1>
} else if (event is SecretLeaseCreatedEvent && event.mode == ROTATE) { // <2>
    val credentials = event.credentials // <3>
    // TODO Update database connection
}

<1> The rotating secret is requested
<2> The new secret event is a rotating 

SecretLeaseCreatedEvent

<3> The event contains the new database credentials

The 

SecretLeaseCreatedEvent
 contains the new credentials requested from Hashicorp Vault. The 
event.credentials
 property is an extension property (see code below).

Details of extracting the secrets safely

The 

SecretLeaseCreatedEvent
 contains a 
Map<String, Object>
 with the secrets, so there is no typesafe option to get the database credentials. If for some reason the event does not contain the credentials we are again in the situation, that we cannot contact the database anymore. In that case I would prefer to shut down the application. That’s why we need the
ConfigurableApplicationContext
 to shut down the Spring application.

Let’s add this as another autowired dependency to this class:

@Configuration
class VaultConfig(
        private val leaseContainer: SecretLeaseContainer,
        @Value("\${spring.cloud.vault.database.role}")
        private val databaseRole: String
) {

Now we can extract the credentials from the event. The extension property

event.credentials
 returns 
null
 if the credentials cannot be received. With the 
ConfigurableApplicationContext
 we can handle this error case:

if (credentials == null) {
    log.error { "Cannot get updated DB credentials. Shutting down." }
    applicationContext.close() // <1>
    return@addLeaseListener // <2>
}
refreshDatabaseConnection(credentials) // <3>

<1> If we cannot get the renewed credentials shutdown the application
<2> because of the return from the lambda, 

credentials
 is smart casted to a non-nullable value after the 
if
 block. Kotlin is awesome!
<3> here the 
credentials
 cannot be 
null
 and can be used to refresh the database connection

Now let’s see how the credentials are retrieved from the event within the extension property:

private val SecretLeaseCreatedEvent.credentials: Credential?
    get() {
        val username = get("username") ?: return null // <1>
        val password = get("password") ?: return null // <1>
        return Credential(username, password)
    }

private fun SecretLeaseCreatedEvent.get(param: String): String? {
    return secrets[param] as? String // <2>
}

private data class Credential(val username: String, val password: String)

<1> username and password are extracted using the extension method 

get()
. If one of the 
get()
 calls return 
null
 then 
null
 is returned instead of a 
Credential

<2> the secret is read out of the map and with 
as?
 safe casted
 to a 
String
. If the entry does not exist in the map or is not a 
String
 then 
null
 is returned

Refresh the database connection

(Refreshed version of access restriction - Image by Nenad Maric from pixabay)

Now that we know the new credentials we have to ensure that these fresh secrets are used instead of the old ones.

private fun refreshDatabaseConnection(credential: Credential) {
    updateDbProperties(credential) // <1>
    updateDataSource(credential) // <2>
}

<1> first update the database system properties
<2> finally update the datasource to use the newly created credentials

To update the datasource credentials we need the 

HikariDataSource
. So, let’s add this also to the constructor:

@Configuration
class VaultConfig(
        private val applicationContext: ConfigurableApplicationContext,
        private val hikariDataSource: HikariDataSource,
        private val leaseContainer: SecretLeaseContainer,
        @Value("\${spring.cloud.vault.database.role}")
        private val databaseRole: String
) {

Utilizing the 

HikariDataSource
 we can update the database credentials used by the Spring application:

private fun updateDbProperties(credential: Credential) { // <1>
    val (username, password) = credential
    System.setProperty("spring.datasource.username", username)
    System.setProperty("spring.datasource.password", password)
}

private fun updateDataSource(credential: Credential) {
    val (username, password) = credential
    log.info { "==> Update database credentials" }
    hikariDataSource.hikariConfigMXBean.apply { // <2>
        setUsername(username)
        setPassword(password)
    }
    hikariDataSource.hikariPoolMXBean?.softEvictConnections() // <3>
            ?.also { log.info { "Soft Evict Hikari Data Source Connections" } }
            ?: log.warn { "CANNOT Soft Evict Hikari Data Source Connections" }
}

<1> Updating the database system properties is technically not mandatory but ensures consistency, if other parts of the system rely these properties being accurate
<2> From the 

HikariDataSource
 we can get the
HikariConfigMXBean
which allows setting the new credentials
<3> As the final step, all connections have to be evicted to use the new credentials

Summary

With these steps the PostgreSQL or other relational database credentials can be rotated, when there Hashicorp Vault leases expire. This works at runtime and without downtime.

The logs will look something like this:

Lease change for DB: (org.springframework.vault.core.lease.event.SecretLeaseExpiredEvent[source=RequestedSecret [path='database/creds/readonly', mode=RENEW]]) : (Lease [leaseId='database/creds/readonly/wzUQ81Ng4YQcBwdAyLrSZSvd', leaseDuration=PT10S, renewable=true])
Replace RENEW for expired credential with ROTATE
Lease change for DB: (org.springframework.vault.core.lease.event.SecretLeaseCreatedEvent[source=RequestedSecret [path='database/creds/readonly', mode=ROTATE]]) : (Lease [leaseId='database/creds/readonly/ur8C5V1wJMSAdiatwkWXCi03', leaseDuration=PT30S, renewable=true])
==> Update database credentials
Soft Evict Hikari Data Source Connections

The complete repository can be found on GitHub.

Finally the handling of expiring Hashicorp Vault database secrets in a Spring application is production-ready.

Originally published at https://secrets-as-a-service.com on February 18, 2020

Image Source