Top 5 Tips for Securing Your GitOps Environments

What is GitOps

GitOps is a software development method in which all aspects of a project, including application code, configuration files, and infrastructure, are stored in a Git repository. When a Git commit is pushed to source control, changes are triggered to update the cloud infrastructure or application.

GitOps uses Git as a single source of truth. This means that only Git-based changes can trigger infrastructure changes, application builds, and deployments. There are no “ad hoc” deployments, and this improves consistency and reliability. GitOps is becoming widely used by organizations that use Kubernetes to orchestrate and manage their microservices applications.

How Does GitOps Improve Container Security

Traditional software development models do not provide robust mechanisms to secure infrastructure and application code. This led to the rise of supply chain attacks in recent years, with devastating global attacks like SolarWinds and Kaseya.

GitOps defines containers and services as code, making infrastructure an integral part of the application. There are several ways this can promote the security of containerized environments and the continuous integration / continuous delivery (CI/CD) infrastructure:

• GitOps creates complete separation between CI and CD, meaning that developers working in the CI environment never have access to the Kubernetes cluster or other production environment.

• When infrastructure and deployment are fully automatic, it is easier to build in mandatory source code scanning and container image scanning.

• Cloud misconfiguration is a common cause of security breaches. GitOps makes it possible to immediately identify misconfiguration or tampering and roll back to a safe configuration.

• GitOps makes it easy to identify containers infected by malware or otherwise modified by malicious parties because they will diverge from the configuration stored in the source code.

• With infrastructure as code (IaC), it is possible to test and verify security for infrastructure and configuration just like you would do for software code.

  
  

5 Tips for Securing GitOps Environments

Embed Security Scanning Early

One of the main benefits of GitOps is that many deployment processes are automated in the pipeline. Developers only need to write code, test it locally, send pull/merge requests, commit their code and move on. GitOps abstracts developers from the process after committing code, so it's important to provide tools for security screening early in the process.

One approach is to incorporate security by integrating static application security testing (SAST) integrated development environment (IDE) and the build process. Another possibility is to expose developers to feedback from unit and integration tests using pre-commit hooks. The sooner a test fails, the less friction in the later stages of the pipeline.

Both the version control system and continuous integration pipeline offer natural places to integrate security scans. These scans should check application source code, container images, and infrastructure as code (IaC) templates.

Separate the Build and Deploy Process (CI and CD)

Traditional DevOps practices treated CI/CD processes as one entity, typically served by one pipeline. In a GitOps work model, they are two separate processes:

• The continuous integration process is responsible for building the application code and packaging it into container images. This part stays the same when an organization transitions to GitOps.
• The continuous deployment process changes in a GitOps environment. It is fully automated, adjusting the Kubernetes cluster to match the desired state described in the Git repository. As soon as code from the master branch is successfully merged, the cluster state is reconciled with the new desired configuration.

A common GitOps approach to isolating these processes is to deploy an additional Git repository as an intermediary between CI and CD. This repository contains information about the environment, and each commit triggers the deployment process. In this way, the CI process doesn't touch the underlying infrastructure like a Kubernetes cluster.

Separating the deployment pipeline from the deployment pipeline provides strong protection against configuration errors and helps improve security and compliance. It also means that any change to the production environment has a clear audit trail because it is exclusively managed through the central Git repository.

Use Container Firewalls

Because containers are constantly started and stopped, it is difficult to monitor and secure container traffic at the network layer. Traditional network firewalls not only struggle to see east-to-west traffic within and between hosts but also cannot keep up with the constant changes that occur in a containerized environment.

Container firewalls provide nearly the same level of protection as next-generation firewalls (NGFW) deployed at the edge. They can filter and control all container traffic, including east-west, north-south, and container-to-container traffic.

A cloud-native container firewall isolates and protects workloads, application stacks, and services, even as containers scale up and down across multiple hosts. Additionally, container firewalls can help protect ingress and egress traffic for a Kubernetes cluster.

Be Strict About Development Policies

GitOps uses trunk-based development. One of the problems with this approach is that you are developing in a single branch, which becomes a single point of failure. It might be concerning to use a single branch not only for your production environment but for the entire development organization.

A way to protect the main branch is GitHub branch protection (or a similar feature provided by other Git systems). Branch protection prevents someone from forcibly changing the trunk, and also prevents branches from being accidentally or intentionally deleted.

In addition, you should set clear and strict rules for who can perform a merge and when. For example, you can require a minimum number of approvals from a trusted pool of senior developers.

Prevent Cloud Drift

Any changes to your cloud environment should be included in an Infrastructure as Code (IaC) template stored in a central Git repository. These templates need to be synchronized with the cloud environment using coordination loops and immutable compute instances.

In a pure GitOps model, the GitOps controller continuously matches the desired state of the IaC template with the actual state of the production resource and updates the resource when deviations are detected.

But many organizations do not use a pure GitOps model, meaning that some computing resources are not immutable and it may not be possible to remove and replace them at will. Maintenance efforts or other changes to resources then lead to configuration drift.

Be aware of cloud drift and ensure you have full visibility and monitoring to ensure accountability. Once the drift is identified, either undo the changes or add them to the IaC template and push them to the pipeline to ensure they become part of the GitOps audit trail.

Conclusion

In this article, I explained the basics of GitOps, showed how GitOps improves security for CI/CD pipelines and provided best practices to help you secure your GitOps environments:

1. Separate the build and deploy process to ensure developers who have access to CI systems cannot alter production systems.
2. Use container firewalls to limit traffic between containers and pods, and control ingress/egress traffic.
3. Embed security scanning early to ensure container images and IaC templates do not contain vulnerabilities or malicious components.
4. Set strict development policies to ensure everyone uses the GitOps model in a consistent way.
5. Prevent cloud drift by identifying ad hoc changes that conflict with the GitOps process and eliminating them.