In this article, we will look into common ways to secure secrets in a Kubernetes application and how to manage them in a GitOps workflow based on ArgoCD with the help of Sops.

The problem is the following: your application depends on some secrets that you need to store securely and make available to your running application.

You can address this requirement in two ways:

You put these secrets in remote secret manager, for instance, Vault or AWS Secret Manager and you use the provided API or convenient tools like External Secrets
You keep these secrets as vanilla Kubernetes Secret objects, you commit those in your Git repository with your code but you take care of encrypting them with for example sops

This second solution has a clear advantage: you can provide your own GPG key and you don’t need to rely on a cloud provider or any external tools. If your goal is a multi-cloud strategy, it’s the way to go.

If you are using ArgoCD to deploy our Kubernetes objects you may wonder how to integrate Sops with ArgoCD. Let’s see what the ArgoCD documentation has to say.

ArgoCD Stance on Secrets Management

ArgoCD documentation makes it quite clear:

Argo CD is un-opinionated about how secrets are managed. There’s many ways to do it and there’s no one-size-fits-all solution.

Basically, you are left to your own devices to make sops work with ArgoCD. In this article, we will take a look at how we can implement secret handling in an elegant, non-breaking way.

Our Tools of Choice

Let’s recap the tools we will use:

Secrets will be encrypted and decrypted using sops and we will provide our own GPG key as the encryption key
Our app is packaged using Helm
Instead of dealing with sops directly, we will use Helm Secrets

Helm Secrets is essentially a wrapper for Helm that encrypt and decrypt secrets on the fly for you. While no longer under heavy development, it’s still working really well.

But the problem is that ArgoCD doesn’t know this plugin as it only comes with the basic Helm binary built-in. Let’s address this now.

Creating our own Custom ArgoCD

We will create a new ArgoCD docker image which contains exactly what we need:

GPG which handle the encryption key
Sops for encryption/decryption
Helm Secrets plugin

The Dockerfile will look like this:

FROM argoproj/argocd:v1.7.6

ARG SOPS_VERSION="v3.6.1"
ARG HELM_SECRETS_VERSION="2.0.2"
ARG SOPS_PGP_FP="141B69EE206943BA9A64E691A00C9B1A7DCB6D07"

ENV SOPS_PGP_FP=${SOPS_PGP_FP}

USER root  
COPY helm-wrapper.sh /usr/local/bin/
RUN apt-get update && \
    apt-get install -y \
    curl \
    gpg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    curl -o /usr/local/bin/sops -L https://github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux && \
    chmod +x /usr/local/bin/sops && \
    cd /usr/local/bin && \
    mv helm helm.bin && \
    mv helm2 helm2.bin && \
    mv helm-wrapper.sh helm && \
    ln helm helm2 && \
    chmod +x helm helm2

# helm secrets plugin should be installed as user argocd or it won't be found
USER argocd
RUN /usr/local/bin/helm.bin plugin install https://github.com/zendesk/helm-secrets --version ${HELM_SECRETS_VERSION}
ENV HELM_PLUGINS="/home/argocd/.local/share/helm/plugins/"

A few things to note:

First, we set up GPG and Sops
Then we install the Helm Secrets plugin
Finally, we move the ArgoCD default Helm binary as helm.bin and replace it with a wrapper script

This wrapper script will look after the GPG key (you can mount it as a secret volume for example) and if found will import it. Then it will replace every call to helm with calls to helm secrets . Well, that’s in theory because Helm secrets does not support all Helm commands and is quite talkative so we need to alter the output for certain commands.

#! /bin/sh

GPG_KEY='/home/argocd/gpg/gpg.asc'

if [ -f ${GPG_KEY} ]
then     
    gpg --quiet --import ${GPG_KEY}
fi

# helm secrets only supports a few helm commands
if [ $1 = "template" ] || [ $1 = "install" ] || [ $1 = "upgrade" ] || [ $1 = "lint" ] || [ $1 = "diff" ]
then 
    # Helm secrets add some useless outputs to every commands including template, namely
    # 'remove: <secret-path>.dec' for every decoded secrets.
    # As argocd use helm template output to compute the resources to apply, these outputs
    # will cause a parsing error from argocd, so we need to remove them.
    # We cannot use exec here as we need to pipe the output so we call helm in a subprocess and
    # handle the return code ourselves.
    out=$(helm.bin secrets $@)
    code=$?
    if [ $code -eq 0 ]; then
        # printf insted of echo here because we really don't want any backslash character processing
        printf '%s\n' "$out" | sed -E "/^removed '.+\.dec'$/d"      
        exit 0
    else
        exit $code
    fi
else
    # helm.bin is the original helm binary
    exec helm.bin $@
fi

Our ArgoCD image now understands Helm secrets without any additional configuration! Cool.

Our Test Application

Our test application is a Helm chart with encrypted secrets. Please note the

secrets.yaml

file which is supposed to contain sensitive data.

testapp
├── Chart.yaml
├── charts
├── secrets.yaml
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   └── serviceaccount.yaml
└── values.yaml

We encode the secrets with sops using our private key, the same one that is looked after by our Helm wrapper script:

sops -i --encrypt testapp/secrets.yaml

Then we push this Helm chart in our Git repository. The only thing left to do is to create the corresponding Application CRD to watch this new repository:

apiVersion:  argoproj.io/v1alpha1
kind:  Application
metadata:  
  name:  testapp 
  namespace:  argocd
spec:  
  project:  default
  source:  
    repoURL: [email protected]:my/repo/charts.git   
    targetRevision:  master
    path:  charts/testapp
    helm:
      releaseName: testapp
      valueFiles:
        - "secrets.yaml"
  destination:  
    server:  https://kubernetes.default.svc
    namespace:  testapp
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

After

kubectl apply -f

this, you should see your new application appears in the ArgoCD dashboard. Secrets have been decrypted under the hood using the provided GPG key and the app is working properly.

From an ArgoCD standpoint, the Helm wrapper appears as the built-in Helm binary so any GUI functionalities related to Helm are still working as usual.

ArgoCD Plugin as an Alternative Solution

To be exhaustive, let’s mention a simpler solution to our problem. We could have used an ArgoCD plugin. A plugin responsibility is to output some YAML that ArgoCD will then send to the Kubernetes API.

To make this work, you will still need a custom ArgoCD Dockerfile but you will not replace the Helm binary, only adding sops and Helm secrets. Then you will declare the plugin. The example below is an extract of the

values.yaml

file using the ArgoCD Helm chart:

server:
  config:
    configManagementPlugins: |
      - name: helmSecrets
        init:   
          command: ["gpg"]  
          args: ["--import", "/home/argocd/gpg/gpg.asc"] # is mounted as a kube secret
        generate:
          command: ["/bin/sh", "-c"]
          args: ["helm secrets template $HELM_OPTS $RELEASE_NAME ."]

It is the same philosophy: we import the GPG key when initializing the plugin and then we call helm secrets template with parameters from the environment to generates the expected YAML objects.

To use the plugin in an Application, do it like this:

piVersion:  argoproj.io/v1alpha1
kind:  Application
metadata:  
  name:  testapp 
  namespace:  argocd
spec:  
  project:  default
  source:  
    repoURL: [email protected]:my/repo/charts.git   
    targetRevision:  master
    path:  charts/testapp
    plugin:
      name: helmSecrets
      env:
        - name: HELM_OPTS
          value: "secrets.yaml"
        - name: RELEASE_NAME
          value: "testapp"
  destination:  
    server:  https://kubernetes.default.svc
    namespace:  testapp
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

You should get the same result as with our previous solution but with one notable exception: ArgoCD cannot recognize your plugin is in fact Helm in disguise so any GUI functionalities related to Helm will not be available, like seeing the values and parameters.

For this reason, our initial solution, albeit a little more complicated, is clearly superior.

Concluding Thoughts

With some quick adjustments, you can make your secrets handling tools work with ArgoCD. Even if ArgoCD doesn’t handle secrets by itself, it is really well thought since you can integrate your own tools quite easily.