Today, I would like to discuss secrets and GitOps in the K8S world. Some of my friends who are not DevOps/SRE engineers and most developers asked me too many times how to store Secrets/API keys/environment variables within the deployment process. I hope this article will help to understand with a trivial explanation of the basics of how-to store secrets in the deployment process in the GitOps Way. First things first Plan of our journey: a small introduction about GitOps deploy k8s cluster ( EKS ) encrypt sensitive data ( SOPS ) deploy ArgoCD ( GitOps tool ) deploy our application in the GitOps way inference GitOps and why it is so important Let's imagine you have some Git repo and deploy your application into Kubernetes via Jenkins, AKA IaC style (Infrastructure as a code). At first look, this architecture is magnificent. We have stored our code in repo, and we deploy it via any CI\CD tool. But why is GitOps so major? The answer to that question lies in how Kubernetes API works. When you apply something into Kubernetes, the API will make you aware that the syntax is acceptable or not. If it's OK, it will give you the lowdown that your resources are declaratively written. Kubernetes will deploy everything you declared. But the first problem is that k8s does not guarantee that and the second one is that if somebody deletes/change/update information about any component or resource, you will never know until it is late. The GitOps way implies that what is in the repo will be in Kubernetes. It literally syncs the states between your repo and Kubernetes. If an unspecified person changes something in Kubernetes ( delete or update config map, for example ), the GitOps tool will sync state and overwrite changes. It works in both ways. It's easy with EKS I will cover the deployment process with tool. I assume that you are already familiar with AWS and with K8S, so I will not explain much on it. First of all, you need to install eksctl. eksctl brew install eksctl For security reasons, it is not a best practice to use AWS access keys. That's why we are going to use IAM Roles to have the ability to use the AWS KMS service within the Kubernetes pods. Hence, we need to create an AWS IAM Policy for KMS keys and assign this Policy to our EKS Worker group, which we will discuss later in this article. I have prepared two git repo for this reason. You are free to use them. So what we are going to do next is: Create IAM role and put ARN reference into the EKS config Create KMS key Deploy EKS cluster with KMS Policy git https://github.com/n-g /argocd.git argocd/eks aws iam create-policy --policy-name AmazonKMSFullAccess --policy-document file://kms_policy.json aws kms create-key --tags TagKey=ENV,TagValue=DEV --description aws kms create-alias -- -name /gitops --target-key-id 48161235-0e48-4aac-9a2c-9b2f99d9e167 vim eks.yaml eksctl create cluster -f eks.yaml clone -s cd "GitOps KMS key" alias alias # Optional, but if you would like to have clear visibility, you should do that # Change line 27 to your AWS account. You should see something like this in the KMS section of the AWS console. How to encrypt the data with SOPS In this section, we will discuss how to encrypt the data with SOPS. I've drawn a small explanation of how works with encryption and the KMS keys. In a few words, SOPS is a tool written by Mozilla to encrypt YAML\JSON and other formats by encryption only values and not keys. More information about this tool you can find . SOPS here To install SOPS, you need to follow this command. We will encrypt with SOPS the official Jenkins helm chart values.yaml and push it to the repo. brew install sops git https://github.com/n-g /argocd-example.git argocd-example/charts SOPS_KMS_ARN= vim values.yaml sops -e values.yaml > values_encrypted.yaml git add . git commit -S -m git push clone -s cd export "arn:aws:kms:us-east-1:591415181778:key/48161235-0e48-4aac-9a2c-9b2f99d9e167" # Change the password on line 106 "Added Jenkins Helm chart with encrypted values file" ArgoCD - Declarative GitOps CD for Kubernetes is a tool which we will deploy into Kubernetes and which will do our deployment process. I've prepared a simple schema of how ArgoCD will do the job. ArgoCD Argo CD is un-opinionated about how secrets are managed and let's you decide how to handle this problem independently by allowing you to customize the deployment workflow with any tool. You can read more about it , but in a few words, you need to do two things: here 1. It would be best if you injected the tool via custom tooling. You will have two options: Adding Tools Via Volume Mounts Build Your Own Image ( which I ) choose ARG ARGOCD_VERSION=v1 FROM argoproj/argocd:${ARGOCD_VERSION} ARG SOPS_VERSION= # Switch to root the ability to perform install USER root # Install tools needed your repo-server to retrieve & decrypt secrets, render manifests # (e.g. curl, awscli, gpg, sops) RUN apt-get update && \ apt-get install -y \ curl \ awscli \ gpg && \ apt-get clean && \ rm -rf / /lib/apt/lists .7 .8 "v3.6.1" for for var /* /tmp/* /var/tmp/* && \ curl -o /usr/local/bin/sops -L https://github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux && \ chmod +x /usr/local/bin/sops # Switch back to non-root user USER argocd 2. Register a new plugin in argocd-cm ConfigMap apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/name: argocd-cm app.kubernetes.io/part-of: argocd name: argocd-cm data: configManagementPlugins: | - name: sops init: command: ["sh", "-xc" ] args: ["sops -d values_encrypted.yaml > values_decrypted.yaml"] generate: command: ["sh", "-xc" ] args: ["helm template -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} -f values_decrypted.yaml ."] Let's deploy the ArgoCD with our . changes kubectl create ns argocd kubens argocd # change namespace kubectl apply -f https://raw.githubusercontent.com/n-g-s/argocd/main/manifests/install.yaml kubectl port-forward svc/argocd-server -n argocd 8080 :443 brew install argocd # cli tool kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2 # Initial password argocd login 127.0.0.1:8080 # user admin and password argocd account update-password # change password to a new one Now you can login with your new password into UI via your Web browser Last but not least - Application deployment! Full description of declarative you can read from , but the only thing I would like to mention. official documentation plugin - sops inject environment variables passed to the plugin So let's deploy our and check if the password is correct with our encryption. application kubectl apply -f https://raw.githubusercontent.com/n-g /argocd-example/main/application.yaml -s kubectl port-forward service/jenkins 8090:8080 -n jenkins Inference As you can see from above, the secured deployment process is not rocket science. There a variety of options on how to do that and how to manage it. The conclusion from my side is - don't be afraid of trying something new. I hope this article explained the basics of GitOps and secure deployment. Few links to read: - Alternative CD tool FluxCD - there are a lot of them. List of Encryption tools
