I Found A Painless Way To Manage Secrets In Google Kubernetes Engine

When deploying complex SaaS platforms, secrets can quickly become the bane of one’s existence. I once wrote about how we ran out of space for our environment variables (in Elastic Beanstalk) because we had so many API keys and secrets. But those were the bad old days.

Things got better after we discovered Berglas. It’s an open-source tool that interfaces with Google Cloud services while at the same time abstracting away all the complexity. Under the hood, it uses the Key Management Service (KMS) to encrypt secrets and Storage Buckets to store them. Alternatively, it can also use the new Google Secret Manager, but I’ll get to that later.

It’s secure. It’s simple. It’s easy to use. I’m going to show you exactly how easy in a minute.

After that, I’ll briefly talk about the other options and why we’re not keen on them — these are options such as:

• Kubernetes Secrets: The built-in Kubernetes solution has a number of drawbacks.
  
• Third-party tools like Vault: Powerful but often very complicated to administer.
  
• Google Secret Manager: The slick newcomer which Berglas also supports — but we don’t like the console UI.

As I said, we’ll get to them later but first, let’s learn more about Berglas.

How does Berglas work with GKE?

Getting Berglas to work with Kubernetes is perhaps slightly trickier than the other Berglas/Google integrations (Cloud Run, App Engine, and so on).

The following illustration shows how it works once we had Berglas all set up

Notice that it works with secret “pointers” that begin with “berglas://”. These are pointers to where the “actual” secrets live in cloud storage.

When we deploy to a cluster, we define environment variables for our secrets, but we populate them with pointers rather than the actual plaintext secrets.

At runtime, the container uses the pointers to get the real secrets from storage. These secrets are decrypted using the Google Key Management Service.

How do you manage secrets with Berglas?

Once you’ve gotten the Kubernetes setup out of the way, secret management is pretty easy. You install the Google Cloud SDK and Berglas CLI to your local environment and bootstrap the resources that Berglas needs. The bootstrap command looks like this:

berglas bootstrap \
- project acme-webapp - bucket secrets-gke-dev - bucket-location europe-west3

The Google Cloud project “acme-webapp” already exists but the bucket is new — this command will create a bucket called “secrets-gke-dev” for you and assign it to the region “europe-west3”. And it will also set up the KMS Key so you don’t have to worry about how to encrypt or decrypt secrets.

When you want to create a secret, you would run a command like this:

berglas create \ 
secrets-gke-dev/magic_password "our little secret" --key projects/acme-webapp/locations/global/keyRings/berglas/cryptoKeys/berglas-key

The value is encrypted locally then sent via TLS to the bucket “secrets-gke-dev” in Google Cloud Storage. On the way to the bucket, the local key is encrypted again using a remote KMS key. You can find more information about the Berglas threat model in the documentation, but I’ve also tried to summarize it in the following illustration.

So as I said earlier, it’s highly secure while also being quite easy to use.

By the way, if you’re storing your keys in Secret Manager, the command is even simpler:

berglas create acme-webapp/magic_password "our little secret"

Setting up Berglas for Kubernetes

One thing I didn’t explain in the first diagram is how containers know what to do with references that begin with “berglas://”.

If you were testing locally, you spawn an application with the “berglas exec” command or pull the Berglas docker image, but this doesn’t scale so well in Kubernetes.

We need to dynamically configure the pods to use Berglas. To do this, we use something called a “mutating webhook”.

The following illustration summarizes how it works:

In our case, it adjusts the YAML configuration for a pod before that pod is admitted to our cluster. It updates the pod configuration so that the containers can understand Berglas references.

To set up a “Mutating Webhook”, you need to give it a configuration — “Mutating Webhook Configuration”. Woohoo! a configuration for editing configurations.

You can deploy that configuration wherever you want as long as it’s accessible by the Kubernetes API server. The Berglas examples show how to deploy a configuration to Google Cloud Functions, which was good enough for us. So far so good.

However, the instructions for setting up Berglas for Kubernetes contains a few gotchas.

Gotchas when setting up Berglas for GKE

Here are a few of the gotchas that we noticed when following the instructions to set up the Berglas Kubernetes integration. These are mentioned in the documentation but you need to be paying close attention. Here’s what we learned by making mistakes

Make sure you have the right regional settings

The most basic example provided for the Berglas bootstrap command doesn’t contain the location flag (it’s described later in an optional step).

So if you go with the basic command “berglas bootstrap — project p.. — bucket b…” you’ll get a multi-regional bucket. However our buckets and our clusters are all assigned to “europe-west3”, so we needed to add the location flag when creating buckets.

The sample applies when deploying the mutating webhook configuration. The documentation provides the following example:

gcloud functions deploy berglas-secrets-webhook \
 -- project ${PROJECT_ID} \
 -- runtime go111 \
 -- entry-point F \
 -- trigger-http

However, this will try to create a webhook in a default region (“us-central1” the last time I looked) that doesn’t necessarily match the one you’re using for your cluster (“europe-west3” in our case).

This mismatch will prevent containers from fetching secrets later on. When we made this mistake, all we could find were “connection refused” errors in the log files — no hint that the region was wrong.

gcloud functions deploy berglas-secrets-webhook \
 --project ${PROJECT_ID} \
 --runtime go111 \
 --entry-point F \
 --trigger-http \
 --region europe-west3

The region is also important when extracting the cloud function URL, so make sure you include it in the command like this:

ENDPOINT=$(gcloud functions describe berglas-secrets-webhook 
—-project ${PROJECT_ID} —-region europe-west3 —-format ‘value(httpsTrigger.url)’)

Make sure that you add a command to the container spec

This is already mentioned in the Berglas documentation (at the very end of the page) I thought it’s worth mentioning more prominently since it tripped us up. Not all Docker containers need to be run with an entry point command (for example PHP containers), but you must define one anyway. if you want the “mutator” to mutate container specifications. In the following example, taken from the Berglas docs, the command is set to [“/bin/envserver”],

spec:
      serviceAccountName: envserver
      containers:
      - name: envserver
        image: sethvargo/envserver
        imagePullPolicy: Always
        command: ["/bin/envserver"]
        env:
        # Example of Berglas with Secret Manager storage.
        - name: API_KEY
          value: sm://PROJECT_ID/my-secret

        # Example of Berglas with Cloud Storage storage.
        - name: API_KEY
          value: berglas://BUCKET_ID/my-secret

OK, but what about the other options for managing secrets?

As promised I wanted to talk a bit about the other options such as Kubernetes built-in secrets, other third-party providers, and Google’s new solution Google Secret Manager.

I’ll start with the latter option because it’s the one that’s most exciting.

What’s Google Secret Manager?

It’s a specialized storage location specifically designed for secrets (unlike buckets which you can use to store just about anything). On the front end, there’s also a fancy new UI.

Google Product managers Seth Vargo and Matt Driscoll announced it on January 22nd with a blog post Introducing Google Cloud’s Secret Manager.

Previously, Berglas sent secrets exclusively to Google Storage Buckets (with encryption provided by KMS). Now, you also have the option of sending your secrets to Secrets Manager instead (KMS encryption is built into Secret Manager).

Why we’re still hesitant to store secrets in Secret Manager

We’ve been using buckets and we’re going to stick with them for the time being.

To understand why, let’s take a brief look at how we organize our secrets right now.

We have “a bucket of secrets” for each of our dev, staging and production clusters. In each bucket, we organize our secrets into “folder” objects that correspond to different components of the web application (backend, frontend, and so on). This means that we can separate the permissions by component. For example, backend developers can only change backend-related secrets.

Technically, “folders” don’t really exist as a concept in the Cloud Storage backend — everything is an object. But parenting “secret objects” to “folder objects” allows us to do pattern matching on the object paths when setting permissions. For example, “grant Chad access to all objects under the path secrets-gke-dev/backend/*”.

This is very simple to do with the gsutil command-line tool.

OK, how do you do it the Secret Manager?

In the Secret Manager, there are no buckets or folders. Everything is stored in one gigantic flat list. To reproduce our setup for a specific project, we would have to do something like in the following screenshot.

In the Berglas CLI, the command for granting access to a secret looks like this:

berglas grant ${PROJECT_ID}/foo --member user:[email protected]

There doesn’t seem to be any obvious way to do pattern matching or batch editing permissions on the secret label. You could perhaps use the Google Cloud CLI or do something clever with one of the client libraries, but for us at least, there’s no tangible benefit for investing the time. Using Berglas with Google Storage buckets suits us just fine.

This doesn’t mean it’s a bad choice. If you’re starting a new project and you don’t have that many secrets why not give it a try? The Berglas FAQ makes the same suggestion.

Q: Should I use Berglas or Secret Manager?

Berglas is compatible with Secret Manager and offers convenience wrappers around managing secrets regardless of whether they reside in Cloud Storage or Secret Manager. New projects should investigate using Secret Manager directly as it has less operational overhead and complexity, but Berglas will continue to support Cloud Storage + Cloud KMS secrets.

One note though, I don’t entirely understand why the FAQ presents a choice between Berglas OR Secret Manager. I understand them to be complimentary.

At the time of writing, the Berglas repo presents itself as “A tool for managing secrets on Google Cloud” with a link to Secret Manager.

Perhaps the confusion exists because Berglas existed before Secret Manager came on the scene in January this year. But according to product manager Seth Vargo’s comments on Twitter, it looks like Berglas was an early “piece” of secret manager

Google launched Secrets Manager in January to provide us all with a dedicated place to store secrets for projects (instead of using Google Storage buckets). But it looks like Google could do more to explain how the components of their secrets management solution have evolved (i.e. Berglas CLI vs Secret Manager web UI vs client libraries).

Until then, the developers and I will continue to argue about whether they’ve been using Secret Manager the whole time under the guise of “Berglas”. Maybe they should have just called it “Secret Manager Client” instead of naming it after that mysterious magician dude. But that would be less exciting.

Anyway, I’m happy that it exists now. One nice thing is that you automatically get a Secret Manager for each of your projects — you don’t have to explicitly provision any “bucket o’ secrets”. It would just be nice if they improved the UI to manage a larger volume of secrets.

So that’s the Secret Manager out of the way. What about…

Third-party secret management providers

Before we discovered Berglas, we were using the Hashicorp Vault to encrypt our secrets. Vault is a powerful, sophisticated tool but it’s not exactly easy to use. And you still have to store your encrypted secrets somewhere. We put our encrypted secrets in GitHub because we need to have them versioned.

This is a problem especially tough for early-stage startups who don’t yet have anyone who has experience in cryptography and key management. This became evident when we were helping a startup and had to introduce Vault.

To make it easier, we wrote a whole bunch of bash scripts for the developers to use whenever they want to create new secrets. However, it also required that they announced their intention to change or add secrets in a special Slack channel. This was an extra measure to prevent anyone else from editing keys at the same time and potentially corrupting the secrets. But it was by no means an ideal solution.

Mozilla SOPs and Bitnami’s Sealed Secrets for Kubernetes are also popular solutions but again they have too many options. We needed a simple tool that developers can use which abstracts away all the complexity related to encryption and key management. As I said before, early-stage startups often don’t have the time and resources to invest in more complex key management solutions.

Kubernetes Secrets

Kubernetes also offers its own solution for securely storing secrets. You can mount secrets as custom data volumes or expose them as environment variables which your containers can access. This solution might be OK if you’re only using Kubernetes — but we need a solution for managing secrets outside of Kubernetes too. It’s much easier to have one centralized solution for all “secret management” use cases. Also, remember that we also need to version control for our secrets and Kubernetes Secrets does provide any way to that automatically.

The Kubernetes secrets solution isn’t the easiest to use either. Secrets aren’t encrypted by default —you need to base64 encode the before put them in your configuration file. You need to explicitly enable encryption so that you’re not storing plaintext secrets into etcd. If you are working with a lot of secrets, it becomes unwieldy to access them as mounted volumes. This might tempt you into the unsavory option of mounting your secrets as plaintext values in environment variables. I won’t go into detail on why this is a bad idea, but needless to say, your secrets can easily be leaked if they’re stored as plaintext in environment variables — for example, dumped into log files or accessed by third-party code.

Summary

Whether you use Cloud Storage Buckets or the new Secrets Manager component, Berglas is a great tool for managing your secrets in Google Cloud. The primary advantage for us is that we have fine-grained control over who can access different secrets. This is essential when administering multiple projects with different teams of developers. We think this admin work is easier to do with Google Storage buckets, but there’s no reason why you couldn’t do it with Secrets Manager as well. And for the developers that we support, it also simplifies the process of creating secrets. No more custom bash scripts and special Slack channels. Essentially, Berglas helped us fulfill our primary mission as an operational VC: clearing away operational obstacles so that developers can focus on building great products.

A Disclosure Note On The Author and Project A

Merlin blogs about developer innovation and new technologies at Project A — a venture capital investor focusing on early-stage startups. Project A provides operational support to its portfolio companies, including developer expertise. As part of the IT team, he covers the highlights of Project A’s work helping startups to evolve into thriving success stories.

Previously published at https://insights.project-a.com/a-painless-way-to-manage-secrets-in-google-kubernetes-engine-3100eb35a313?source=friends_link&sk=9bf900fc395429bbf1af70b223d68155