Managing Multi-Environment Kubernetes Deployments with GitOps and Argo CD

Go through any GitOps tutorial (including some written by myself) and you will learn how to set up a single environment that works perfectly with your GitOps tool of choice. However, in reality, you will most likely need more than a single environment. You might have a development environment, a staging environment, and a production environment. How to promote a release from development, to staging, and finally to production, is something that most tutorials gloss over.

In this post, I will sketch out a possible solution for solving environment promotions. This post takes a lot of inspiration from a great Codefresh article but tries to fill in details of how exactly to do this using GitHub Actions and Helm-charts instead of plain Kubernetes manifests with some Kustomize overlays.

To keep this post light and avoid all of the production-scenario complications I will restrict myself to the following:

• I will not include an actual application with source code and a continuous integration pipeline.
• I will include the configuration repository (config repo) which contains Kubernetes manifests in the shape of a simple Helm chart.
• I will restrict myself to using three environments: development, staging, and production.
• I will not discuss the need for hot fixes directly into production, I will assume all the releases go through development, to staging, and finally to production.
• I will start off with a working Kubernetes cluster with Argo CD installed in the argocd namespace. Tutorials on how to create a Kubernetes cluster and install Argo CD is available online.
• I will use the same Kubernetes cluster for all my environments, but I will separate them into their own namespaces.
• I will use the same Argo CD project (the one named default) for all my environments. I will not include any additional Argo CD concepts such as RBAC.
• My Argo CD applications will all just care about the main branch in my repository. There is no need to separate the different environments into different branches if you follow the structure outlined in this post.

Configuration repository

My application consists of a simple Helm chart with a single Kubernetes deployment. The application could theoretically be as complicated as it must be, but I don't want to spend too much time explaining what the application consists of.

The structure and content of the repository look like this:

.
├── .github
│   └── workflows
│       ├── promote-to-production.yaml
│       └── promote-to-staging.yaml
├── app
│   ├── Chart.yaml
│   ├── environments
│   │   ├── development
│   │   │   ├── values.yaml
│   │   │   └── version.yaml
│   │   ├── production
│   │   │   ├── values.yaml
│   │   │   └── version.yaml
│   │   └── staging
│   │       ├── values.yaml
│   │       └── version.yaml
│   └── templates
│       └── deployment.yaml
└── gitops
    ├── development.yaml
    ├── production.yaml
    └── staging.yaml

In this section, I will go through the details of what is in the app and gitops directory. In the next section, I will go through what is in the .github directory.

I will not bother going through the Chart.yaml file for Helm, it is not of particular interest here. The manifest for my deployment is shown in the following code snippet:

# app/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: {{ .Values.deployment.replicas }}
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      name: my-pod
      labels:
        app: my-application
    spec:
      containers:
        - name: webserver
          image: {{ .Values.deployment.image.name }}:{{ .Values.deployment.image.tag }}

There are three things I will vary between my environments:

• The number of replicas in the deployment. This value is expected to be different for each environment, but should not be updated frequently.
• The container image name and tag. The tag value is expected to change frequently.

For each environment I will use two Helm value files, they are located in the corresponding environment directory app/environments/development, app/environments/staging, or app/environments/production:

• values.yaml contains differences between the environments that are not expected to be promoted from one environment to the next. In this case, it just contains the number of replicas in the deployment. For the development environment, these files look like this:

  # app/environments/development/values.yaml
  deployment:
    replicas: 1
  

  
  

• version.yaml contains changes that should be promoted from one environment to the next. In my example, it contains the name and tag of the container image. To start off, the file looks the same for all environments:

  # app/environments/development/version.yaml
  # app/environments/staging/version.yaml
  # app/environments/production/version.yaml
  deployment:
    image:
      name: nginx
      tag: 1.22.0
  

The last set of files in my repository is located in the gitops directory. These are the Argo CD applications for each environment. I would not necessarily include them in the same repository, but in this case, I chose to do that.

The Argo CD application for the development environment looks like the following:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: development-application
  namespace: argocd
spec:
  project: default
  source:
    repoURL: <my github repository url>
    path: app
    targetRevision: main
    helm:
      valueFiles:
        - environments/development/values.yaml
        - environments/development/version.yaml
  destination:
    server: "https://kubernetes.default.svc"
    namespace: development
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The applications for the staging and production environments look similar.

Automate environment promotion with GitHub Actions

So now we have three Argo CD applications, one for each of our environments. To introduce a change in our development environment we would update app/environments/development/version.yaml with a new value for the container name or tag.

More realistically we would update the tag value. To promote this change from development to staging we would simply copy this change from the development environment and push the change:

$ cp app/environments/development/version.yaml app/environments/staging/version.yaml
$ git commit -am "Promote change to staging" && git push

Likewise, to promote the change from staging to production we do another copy operation and push the change:

$ cp app/environments/staging/version.yaml app/environments/production/version.yaml
$ git commit -am "Promote change to production" && git push

This seems so simple that we should be able to automate it! Exactly how you want to automate it will depend a bit on your circumstances, and the kinds of tests and checks you want to make before you promote the change from one environment to the next.

I will show you how to automate it in the following ways using GitHub Actions:

• Any change to the development environment is continuously deployed to the development environment without restrictions. A pull request to promote the change to the staging environment is automatically created.

  
  

• Changes to the staging environment require an approved pull request, once approved the change is deployed to the staging environment. A pull request to promote the change to the production environment is automatically created.

  
  

• Changes to the production environment require an approved pull request, once approved the change is deployed to the production environment.

The workflow to promote a change to the staging environment looks like this:

# .github/workflows/promote-to-staging.yaml
name: Promote to staging

on:
  push:
    branches:
      - main
    paths:
      - app/environments/development/version.yaml

permissions:
  contents: write
  pull-requests: write

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: |
          # configure git client
          git config --global user.email "<email address>"
          git config --global user.name "<name>"

          # create a new branch
          git switch -c staging/${{ github.sha }}

          # promote the change
          cp app/environments/development/version.yaml app/environments/staging/version.yaml

          # push the change to the new branch
          git add app/environments/staging/version.yaml
          git commit -m "Promote development to staging"
          git push -u origin staging/${{ github.sha }}
      - run: |
          gh pr create \
            -B main \
            -H staging/${{ github.sha }} \
            --title "Promote development to staging" \
            --body "Automatically created by GitHub Actions"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The workflow is triggered whenever there is a change to app/environments/development/version.yaml on the main branch. The workflow itself consists of three steps:

1. Check out the source code (since we need to update it!)
2. Configure git, create a new git branch, perform the copy operation, push the change to the new branch
3. Use the GitHub CLI to create a pull request for the new change

We could run additional automation to perform tests in our development environment before we go on to approve the promotion to the staging environment.

Once the change is approved a new workflow is triggered to initiate the promotion to the production environment:

# .github/workflows/promote-to-production.yaml
name: Promote to production

on:
  pull_request:
    types:
      - closed
    paths:
      - app/environments/staging/version.yaml

permissions:
  contents: write
  pull-requests: write

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: |
          # configure git client
          git config --global user.email "<email address>"
          git config --global user.name "<name>"

          # create a new branch
          git switch -c production/${{ github.sha }}

          # promote the change
          cp app/environments/staging/version.yaml app/environments/production/version.yaml

          # push the change to the new branch
          git add app/environments/production/version.yaml
          git commit -m "Promote staging to production"
          git push -u origin production/${{ github.sha }}
      - run: |
          gh pr create \
            -B main \
            -H production/${{ github.sha }} \
            --title "Promote staging to production" \
            --body "Automatically created by GHA"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The difference in this workflow is just the trigger. This workflow is triggered when a pull request has been closed where there were changes to app/environments/staging/version.yaml.

Before we approve the merge request to promote the change to production we can run additional automation against our staging environment. When we are ready we can approve the pull request and the change will be promoted to production.

Note that there is some additional work required to lock down how changes can be promoted. There should be some rules in place that restrict changes directly to the staging and production environments. I might continue my work on this example in future posts because there is a lot more that could be said. For now, I will leave you with this example!

Also published here.