Go through any GitOps tutorial (including ) and you will learn how to set up a single environment that works perfectly with your GitOps tool of choice. However, in reality, you will most likely need more than a single environment. You might have a development environment, a staging environment, and a production environment. How to promote a release from development, to staging, and finally to production, is something that most tutorials gloss over. some written by myself In this post, I will sketch out a possible solution for solving environment promotions. This post takes a lot of inspiration from a great Codefresh but tries to fill in details of how exactly to do this using GitHub Actions and Helm-charts instead of plain Kubernetes manifests with some Kustomize overlays. article To keep this post light and avoid all of the production-scenario complications I will restrict myself to the following: I will include an actual application with source code and a continuous integration pipeline. not I will include the configuration repository (config repo) which contains manifests in the shape of a simple Helm chart. Kubernetes I will restrict myself to using three environments: development, staging, and production. I will discuss the need for hot fixes directly into production, I will assume all the releases go through development, to staging, and finally to production. not I will start off with a working Kubernetes cluster with Argo CD installed in the namespace. Tutorials on how to create a Kubernetes cluster and install Argo CD is available online. argocd I will use the same Kubernetes cluster for all my environments, but I will separate them into their own namespaces. I will use the same project (the one named ) for all my environments. I will not include any additional Argo CD concepts such as RBAC. Argo CD default My Argo CD applications will all just care about the branch in my repository. There is no need to separate the different environments into different branches if you follow the structure outlined in this post. main Configuration repository My application consists of a simple Helm chart with a single Kubernetes deployment. The application could theoretically be as complicated as it must be, but I don't want to spend too much time explaining what the application consists of. The structure and content of the repository look like this: .
├── .github
│   └── workflows
│       ├── promote-to-production.yaml
│       └── promote-to-staging.yaml
├── app
│   ├── Chart.yaml
│   ├── environments
│   │   ├── development
│   │   │   ├── values.yaml
│   │   │   └── version.yaml
│   │   ├── production
│   │   │   ├── values.yaml
│   │   │   └── version.yaml
│   │   └── staging
│   │       ├── values.yaml
│   │       └── version.yaml
│   └── templates
│       └── deployment.yaml
└── gitops
    ├── development.yaml
    ├── production.yaml
    └── staging.yaml In this section, I will go through the details of what is in the and directory. In the next section, I will go through what is in the directory. app gitops .github I will not bother going through the file for Helm, it is not of particular interest here. The manifest for my deployment is shown in the following code snippet: Chart.yaml # app/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: {{ .Values.deployment.replicas }}
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      name: my-pod
      labels:
        app: my-application
    spec:
      containers:
        - name: webserver
          image: {{ .Values.deployment.image.name }}:{{ .Values.deployment.image.tag }} There are three things I will vary between my environments: The number of in the deployment. This value is expected to be different for each environment, but should not be updated frequently. replicas The container image and . The value is expected to change frequently. name tag tag For each environment I will use two Helm , they are located in the corresponding environment directory , , or : value files app/environments/development app/environments/staging app/environments/production contains differences between the environments that are not expected to be promoted from one environment to the next. In this case, it just contains the number of replicas in the deployment. For the development environment, these files look like this: values.yaml # app/environments/development/values.yaml
deployment:
  replicas: 1 contains changes that should be promoted from one environment to the next. In my example, it contains the name and tag of the container image. To start off, the file looks the same for all environments: version.yaml # app/environments/development/version.yaml
# app/environments/staging/version.yaml
# app/environments/production/version.yaml
deployment:
  image:
    name: nginx
    tag: 1.22.0 The last set of files in my repository is located in the directory. These are the Argo CD applications for each environment. I would not necessarily include them in the same repository, but in this case, I chose to do that. gitops The Argo CD application for the development environment looks like the following: apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: development-application
  namespace: argocd
spec:
  project: default
  source:
    repoURL: <my github repository url>
    path: app
    targetRevision: main
    helm:
      valueFiles:
        - environments/development/values.yaml
        - environments/development/version.yaml
  destination:
    server: "https://kubernetes.default.svc"
    namespace: development
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true The applications for the staging and production environments look similar. Automate environment promotion with GitHub Actions So now we have three Argo CD applications, one for each of our environments. To introduce a change in our development environment we would update with a new value for the container name or tag. app/environments/development/version.yaml More realistically we would update the tag value. To promote this change from development to staging we would simply copy this change from the development environment and push the change: $ cp app/environments/development/version.yaml app/environments/staging/version.yaml
$ git commit -am "Promote change to staging" && git push Likewise, to promote the change from staging to production we do another copy operation and push the change: $ cp app/environments/staging/version.yaml app/environments/production/version.yaml
$ git commit -am "Promote change to production" && git push This seems so simple that we should be able to automate it! Exactly how you want to automate it will depend a bit on your circumstances, and the kinds of tests and checks you want to make before you promote the change from one environment to the next. I will show you how to automate it in the following ways using GitHub Actions: Any change to the development environment is continuously deployed to the development environment without restrictions. A pull request to promote the change to the staging environment is automatically created. Changes to the staging environment require an approved pull request, once approved the change is deployed to the staging environment. A pull request to promote the change to the production environment is automatically created. Changes to the production environment require an approved pull request, once approved the change is deployed to the production environment. The workflow to promote a change to the staging environment looks like this: # .github/workflows/promote-to-staging.yaml
name: Promote to staging

on:
  push:
    branches:
      - main
    paths:
      - app/environments/development/version.yaml

permissions:
  contents: write
  pull-requests: write

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: |
          # configure git client
          git config --global user.email "<email address>"
          git config --global user.name "<name>"

          # create a new branch
          git switch -c staging/${{ github.sha }}

          # promote the change
          cp app/environments/development/version.yaml app/environments/staging/version.yaml

          # push the change to the new branch
          git add app/environments/staging/version.yaml
          git commit -m "Promote development to staging"
          git push -u origin staging/${{ github.sha }}
      - run: |
          gh pr create \
            -B main \
            -H staging/${{ github.sha }} \
            --title "Promote development to staging" \
            --body "Automatically created by GitHub Actions"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} The workflow is triggered whenever there is a change to on the branch. The workflow itself consists of three steps: app/environments/development/version.yaml main Check out the source code (since we need to update it!) Configure git, create a new git branch, perform the copy operation, push the change to the new branch Use the GitHub CLI to create a pull request for the new change We could run additional automation to perform tests in our development environment before we go on to approve the promotion to the staging environment. Once the change is approved a new workflow is triggered to initiate the promotion to the production environment: # .github/workflows/promote-to-production.yaml
name: Promote to production

on:
  pull_request:
    types:
      - closed
    paths:
      - app/environments/staging/version.yaml

permissions:
  contents: write
  pull-requests: write

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: |
          # configure git client
          git config --global user.email "<email address>"
          git config --global user.name "<name>"

          # create a new branch
          git switch -c production/${{ github.sha }}

          # promote the change
          cp app/environments/staging/version.yaml app/environments/production/version.yaml

          # push the change to the new branch
          git add app/environments/production/version.yaml
          git commit -m "Promote staging to production"
          git push -u origin production/${{ github.sha }}
      - run: |
          gh pr create \
            -B main \
            -H production/${{ github.sha }} \
            --title "Promote staging to production" \
            --body "Automatically created by GHA"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} The difference in this workflow is just the trigger. This workflow is triggered when a pull request has been closed where there were changes to . app/environments/staging/version.yaml Before we approve the merge request to promote the change to production we can run additional automation against our staging environment. When we are ready we can approve the pull request and the change will be promoted to production. Note that there is some additional work required to lock down how changes can be promoted. There should be some rules in place that restrict changes directly to the staging and production environments. I might continue my work on this example in future posts because there is a lot more that could be said. For now, I will leave you with this example! Also published here.

This story contains new, firsthand information uncovered by the writer.

Managing Multi-Environment Kubernetes Deployments with GitOps and Argo CD

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Delivering Cloud Infrastructure the GitOps Way with Crossplane and ArgoCD

Getting Started with GitOps 2.0 Using ArgoCD & Codefresh

How to Handle Kubernetes Secrets with ArgoCD and Sops

How to Optimize Kubernetes and Argo Workflow Spending

Streamlining Kubernetes App Deployments with ArgoCD: A Step-by-Step GitOps Guide

The Top 5 DevOps Tools and Services to Consider as a Startup

Delivering Cloud Infrastructure the GitOps Way with Crossplane and ArgoCD

Getting Started with GitOps 2.0 Using ArgoCD & Codefresh

How to Handle Kubernetes Secrets with ArgoCD and Sops

How to Optimize Kubernetes and Argo Workflow Spending

Streamlining Kubernetes App Deployments with ArgoCD: A Step-by-Step GitOps Guide

The Top 5 DevOps Tools and Services to Consider as a Startup

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps