Managing Secrets in Docker Swarm

Disclaimer: I wrote this article first for semaphoreci.com community blog, to whom it belongs and it is shared in this blog under by-nc-sa 4.0 licence.

Introduction

Docker changed how development, testing and running software are done, and that’s why software engineers and companies should develop new habits and follow best practices. On the other hand, orchestration technologies like Docker Swarm, K8S or Mesos are also changing depending on the industry demands. Security management, including secret management in Docker Swarm, is one of the things that have evolved as a reaction to these changes.

When working with Docker and orchestration tools, following security best practices is important. Software is becoming more distributed and with the rise of new development models, e.g. Microservices Architecture, communication between software components is adopting new patterns like message-based communication.

In a production environment, containers running different services may exchange some critical information. When deploying a software to different environments, it is evident that storing critical information like production passwords in the development environment is not a good practice. There are multiple scenarios where we can notice the need of a standardized interface for accessing secrets.

Datacenter Operating System, Kubernetes and other orchestration technologies have their own integrated secrets management solutions. In this tutorial, we re going to discover Docker native secret management.

Managing Secrets in Swarm Mode

In this part of this tutorial, we are going to see how to create, attach and rotate sensitive data using Docker Secrets.

How Docker Secrets Work

Starting from the version 1.13, Docker users can use Docker Secrets in a Swarm cluster. Managers in Docker Swarm act as an authoritative delegation to coordinate secrets management.

Secrets could be passwords, keys, certificates, sensitive environment data or any other custom data that a developer wants to protect e.g a database name, the administrator username, etc.

Docker Secrets is only available in the Swarm mode, so standalone containers can not use this feature. Swarm Mode allows you to centrally manage sensitive data and messages by encrypting and transmitting them in a secure way and only to containers that need to access to it ( the principle of least-privilege).

When a user adds a new secret to a Swarm cluster, this secret is sent to a manager using a TLS connection.

Note: TLS is a cryptographic protocol that provides communications security over a network by providing communication encryption, privacy and data integrity.

In order to make all managers up to date with a new secret, when one manager node receives the secret, it saves it to a Raft store with a 256-bit key.

Creating a Secret

In order to see how everything works, let’s create a new Docker machine:

```docker-machine create --driver virtualbox secrets-testing

> Running pre-create checks...> (secrets-testing) Default Boot2Docker ISO is out-of-date, downloading the latest release...> (secrets-testing) Latest release for github.com/boot2docker/boot2docker is v17.05.0-ce> (secrets-testing) Downloading /home/eon01/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v17.05.0-ce/boot2docker.iso...> (secrets-testing) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%> Creating machine...> (secrets-testing) Copying /home/eon01/.docker/machine/cache/boot2docker.iso to /home/eon01/.docker/machine/machines/secrets-testing/boot2docker.iso...> (secrets-testing) Creating VirtualBox VM...> (secrets-testing) Creating SSH key...> (secrets-testing) Starting the VM...> (secrets-testing) Check network to re-create if needed...> (secrets-testing) Waiting for an IP...> Waiting for machine to be running, this may take a few minutes...> Detecting operating system of created instance...> Waiting for SSH to be available...> Detecting the provisioner...> Provisioning with boot2docker...> Copying certs to the local machine directory...> Copying certs to the remote machine...> Setting Docker configuration on the remote daemon...> Checking connection to Docker...> Docker is up and running!> To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env secrets-testing```

You should see the created machine using :

```docker-machine ls

> NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS> secrets-testing - virtualbox Running tcp://192.168.99.100:2376 v17.05.0-ce```

Let’s see the machine variables:

```docker-machine env secrets-testing

> export DOCKER_TLS_VERIFY="1"> export DOCKER_HOST="tcp://192.168.99.100:2376"> export DOCKER_CERT_PATH="/home/eon01/.docker/machine/machines/secrets-testing"> export DOCKER_MACHINE_NAME="secrets-testing"

```

If everything is ok, we can connect the current shell session to the new machine:

```eval "$(docker-machine env secrets-testing)"```

We can now start a Swarm cluster to run our tests. Using the IP address of the created machine, initialize the Swarm:

```docker swarm init --advertise-addr 192.168.99.100```

You should see something similar to:

```> Swarm initialized: current node (sqd5o5vj5c2r9gn53jd1uc4ig) is now a manager.>> To add a worker to this swarm, run the following command:>> docker swarm join \> --token SWMTKN-1-0atbw3c3geeyievi62dkgrw03pauh93rhfz41mdveo21ei1fb6-abf6azeaezaea \> 192.168.99.100:2377>> To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.```

Our manager is created, and you can create new machines and add new managers or workers of your choice.

Now, let’s create a new random Docker secret. While the secret will be created using date |md5sum|awk '{print $1}', a Docker secret is created using this command:

docker secret create [OPTIONS] SECRET file|-

We can run a single command to create a Docker secret:

```date |md5sum|awk '{print $1}' | docker secret create my_secret -> qk4k9adgri6b3ubjua9nca5fp```

The created secret is called my_secret.

When you type, docker secret ls you can get a view of the used secrets:

```> ID NAME CREATED UPDATED> qk4k9adgri6b3ubjua9nca5fp my_secret About a minute ago About a minute ago```

qk4k9adgri6b3ubjua9nca5fp is a random and unique identifier of the recently created secret.

If you want to have more information about my_secret, you can use:

docker secret inspect  my_secret

and you will see a JSON output with different other information such as the creation or the update date.

```> [> {> "ID": "qk4k9adgri6b3ubjua9nca5fp",> "Version": {> "Index": 10> },> "CreatedAt": "2017-05-22T13:25:23.815462834Z",> "UpdatedAt": "2017-05-22T13:25:23.815462834Z",> "Spec": {> "Name": "my_secret",> "Labels": {}> }> }> ]```

Our secret is now stored in the encrypted Raft logs.

Consuming Docker Secrets

Let’s create a new service that will be using the created secret ( my_secret ).

docker service create --name my_test_app --secret my_secret eon01/infinite

eon01/infinite is a Docker image that I created for this kind of tests that uses Alpine to run an infinite loop.

Now, we have a running service named my_test_app using the secret my_secret. We can view it using docker service ls or even docker ps.

```docker ps

> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES> db38ae5f19fa eon01/infinite:latest "/bin/sh -c 'tail ..." 10 minutes ago Up 10 minutes my_test_app.1.cc71i2rm0cl6pghn7z5ht7b98```

Only the created service is using the password, and since our service is running one container, only this container can see the secret. In other words, when a secret is attached to a service, only this service and its tasks have access to this sensitive data. The secret is now stored to a temporary file system (tmpfs) under /run/secrets/my_secret

Tmpfs is the temporary file storage facility on many Unix-like operating systems stored in volatile memory instead of a persistent storage device.

Since our container id is db38ae5f19fa, we can check the tmpfs content using:

```docker exec -it db38ae5f19fa cat /run/secrets/my_secret

> 21ac8406497dded57ceafb7767e53633```

If you create other secrets for the same services, you can find them in the same directory under /run/secrets/<secret_name>

Sharing Docker Secrets

In order to see how sensitive data could be shared between two applications, we can create a new service called my_other_test_app and send it the created secret. In our case, my_other_test_app will use the same password but with a different name (my_other_secret):

```docker service create --name my_other_test_app --secret source=my_secret,target=my_other_secret,mode=0400 eon01/infinite```

We have set the mode to share the secret (0400) but it is possible also to add other options like UID or GID. Let’s get inside the container to verify the content of the create secret:

```docker exec -it 51fe104e416b cat /run/secrets/my_other_secret```

And we will find the same output as the previous command docker exec -it db38ae5f19fa cat /run/secrets/my_secret which is the first secret 21ac8406497dded57ceafb7767e53633.

Rotating Docker Secrets

Rotating sensitive data like passwords is a security best practice and it is possible using the Docker Swarm Secrets feature.

Start by creating a new secret:

```date |md5sum|awk '{print $1}' | docker secret create my_new_secret -```

After that, the old secret should be deleted and updated by the new one and this is done in one command:

docker service update --secret-rm my_secret --secret-add  source=my_new_secret,target=my_secret  my_test_app

If you want to verify, you will find that the old secret (21ac8406497dded57ceafb7767e53633 in my case) has now a new value:

docker exec -it 5152ba89ea7f cat /run/secrets/my_secret

You can also inspect the new secret:

```docker secret inspect my_new_secret

> [> {> "ID": "loame64czkbto3z87s37iy2ds",> "Version": {> "Index": 34> },> "CreatedAt": "2017-05-22T14:03:46.627206229Z",> "UpdatedAt": "2017-05-22T14:03:46.627206229Z",> "Spec": {> "Name": "my_new_secret",> "Labels": {}> }> }> ]```

Conclusion

We have seen a native feature in Swarm, it was introduced in Docker 1.13 after realizing how much important secret management is.

All recent releases of Docker CLI and API support this feature but only in the Swarm mode where secrets are stored in the Raft log of the associated Swarm cluster and stored in a temporary filesystem inside a container.

Secret management with Docker is useful: sensitive data is immutable, never written to a disk and never sent in clear text to the network. Other orchestration technologies like Kubernetes have different models of security and secrets management. If security is important in your project, you should compare these different models and choose the best fit for you.

Connect Deeper

If you resonated with this article, you might be interested in my book: Painless Docker: Unlock The Power Of Docker & Its Ecosystem.

You can also subscribe to DevOpsLinks : An Online Community Of Diverse & Passionate DevOps, SysAdmins & Developers From All Over The World and Shipped: An Independent Newsletter Focused On Containers & Orchestration.

You can find me on Twitter and you can also check other books like SaltStack For DevOps : Extremely fast and simple IT automation and configuration management.

If you liked this post, please recommend and share it to your followers.