Weaveworks

weaveworks

Your Definitive GitOps Guide For DevOps Experts And Newbies

Ever wondered how the magic of seamless software development and deployment happens? Say hello to GitOps – your new best friend in the world of coding wonders! Are you ready to enter into the world of GitOps? Let’s dive in! What is GitOps? GitOps is a term that describes a set of practices that use Git as the source of truth for software development and deployment. GitOps leverages the power of version control to manage the entire lifecycle of applications and infrastructure. is a vital component of DevOps, enabling developers to track, collaborate, and review code changes consistently and reliably. Version control also facilitates continuous integration and delivery, as it allows for automated testing and deployment of code changes. Version control GitOps addresses some of the challenges that traditional workflows face, such as manual intervention, configuration drift, human error, etc. Using Git, enables the desired state of the system to always be defined, documented, and versioned in Git. Core Principles of GitOps GitOps also follows some key principles that make it a compelling approach for modern development and deployment. Some of these principles and benefits are: GitOps uses declarative configuration files to describe the desired state of the system. Declarative being that it is supported by a set of facts not instructions. This makes the system easier to understand, maintain, and reproduce. Declarative: It uses automation tools to balance the current state of the system with the desired state defined in Git. This eliminates human intervention which reduces human error while enabling faster and more reliable deployments. Automated: It ensures that the system can be reproduced in any environment, as long as the same configuration files are used. This helps us produce the same result across different stages and platforms. Reproducible: GitOps allows for easy reversal and recovery, as any change made to the system can be traced back to a specific commit in Git. Reversible: GitOps provides a complete history and audit trail of all changes made to the system, as every change is recorded and tracked in Git. This enables transparency, accountability, and compliance. Auditable: Understanding GitOps Concepts 1. Version Control with Git is a distributed version control system that enables developers to manage code changes collaboratively and efficiently. Git allows developers to create branches, which are separate copies of the code base that can be modified independently without affecting the main branch. Your codebase on git is like a tree, it can have so many branches if a branch is cut off, it doesn't affect the entire tree. Git Branching strategies are important in GitOps workflows, as they define how developers work on different features, fixes, or enhancements, and how they merge them back to the main branch. 2. Infrastructure as Code (IaC) Infrastructure as Code (IaC) is a practice that involves using code or configuration files to define, provision, and manage infrastructure resources such as servers, networks, storage, etc. IaC enables developers to treat infrastructure as software, applying the same principles of version control, automation, reproducibility, etc. IaC plays a crucial role in GitOps, as it allows developers to store the configuration files that describe the infrastructure state in Git, along with the application code. This ensures that the infrastructure is always aligned with the application requirements, and can be deployed or updated automatically. Some various tools and technologies enable IaC, such as Aptible which helps in automating infrastructure management. Declarative Configuration Management Declarative configuration management is a technique that involves using configuration files to specify the desired state of a system or an application, rather than using commands or scripts to instruct how to achieve it. Declarative configuration management simplifies the process of defining and maintaining complex systems or applications by abstracting away the implementation details. Declarative configuration management is widely used in GitOps workflows, as it enables developers to store the configuration files that describe the application state in Git, along with the infrastructure state. This ensures that the application is always deployed or updated according to the configuration files, and any discrepancy between the actual and desired state is detected and corrected automatically. Declarative configuration files are usually written in YAML or JSON formats, which are human-readable and machine-parsable. YAML and files can be used to define various aspects of an application, such as its components, dependencies, resources, environment variables, etc. JSON GitOps in Action: Implementation and Tools GitOps Workflow Lifecycle A typical GitOps workflow consists of four main phases: code development, continuous integration, continuous deployment, and release automation. These phases are illustrated in the following diagram: Code Development Phase: This phase involves developing and testing the application code and the configuration files that define the infrastructure and application state. Developers use Git to create branches, make commits, and perform code reviews. Once the code and configuration files are ready, they are merged into the main branch. Continuous Integration Phase: This phase involves building and testing the application code and the configuration files using automation tools. The tools check for syntax errors, code quality, security vulnerabilities, etc., and run unit tests, integration tests, etc. If the tests pass, the code and configuration files are pushed to a remote repository in Git. Continuous Deployment Phase: This phase involves deploying and updating the infrastructure and application state using automation tools. The tools monitor the remote repository in Git for any changes and synchronize the actual state of the system with the desired state defined in the code and configuration files. If there are any discrepancies, the tools apply the changes automatically or notify the developers for manual intervention. Release Automation Phase: This phase involves releasing and delivering the application to the end-users using automation tools. The tools manage the release process, such as creating release notes, tagging versions, rolling out updates, etc. The tools also enable features such as canary deployments, blue-green deployments, etc., to ensure smooth and safe releases. GitOps Tools and Platforms Some various tools and platforms enable GitOps workflows, such as ArgoCD, Flux, Jenkins X, WeaveWorks, etc. These tools and platforms differ in scope, features, and capabilities, but they all share the common goal of automating GitOps workflows. Some of the features and capabilities that these tools and platforms offer are: Git Integration: The tools and platforms integrate with Git repositories to monitor and synchronize changes to the code and configuration files. They also support various Git operations, such as branching, merging, tagging, etc. IaC Support: The tools and platforms support various IaC tools and technologies, such as Aptible. They also support various cloud providers and platforms, such as AWS, Azure, GCP, Kubernetes, etc. Declarative Configuration Management: The tools and platforms support declarative configuration management using YAML or JSON files. They also support various configuration formats and standards, such as Kubernetes manifests, Helm charts, Customize overlays, etc. Continuous Integration and Testing: The tools and platforms provide continuous integration and testing capabilities using various automation tools and frameworks. They also provide feedback loops and quality gates to ensure code quality and reliability. Continuous Deployment and Synchronization: The tools and platforms provide continuous deployment and synchronization capabilities using various automation tools and mechanisms. They also provide reconciliation loops and health checks to ensure system consistency and stability. Release Automation and Delivery: The tools and platforms provide release automation and delivery capabilities using various automation tools and strategies. They also provide observability and traceability features to ensure system performance and availability. Advantages of GitOps Adoption Enhanced Collaboration and Transparency One of the main advantages of GitOps adoption is that it enhances collaboration and transparency among developers and other stakeholders involved in software development and deployment. By using Git as the single source of truth for both code and configuration files. Git also facilitates collaboration by enabling developers to work on different branches without affecting each other’s work. Developers can also perform code reviews using pull requests or merge requests before merging their changes to the main branch. This ensures that the code quality is maintained throughout the development process. GitOps also provides transparency by recording every change made to the system in Git history. Developers can easily track who made what change when using commit messages or tags. This enables accountability and traceability for every change made to the system. Improved Traceability and Auditing Another advantage of GitOps adoption is that it improves traceability and auditing for software development and deployment. GitOps also provides auditing by creating a complete audit trail of all changes made to the system in Git. Developers can easily verify who authorized, approved, or deployed any change using pull requests or merge requests. This enables compliance and governance for software development and deployment. Scalability and Resource Efficiency It enhances scalability and resource efficiency for software development and deployment. By using automation tools to synchronize the system state with the code and configuration files in Git, GitOps ensures that the system can scale up or down according to the demand and requirements. GitOps also enables resource optimization by using automation tools to deploy or update the system efficiently and consistently. GitOps reduces the need for manual intervention and human error, which can lead to wasted resources or inconsistent results. GitOps also leverages the cloud-native capabilities of platforms such as Kubernetes, which can dynamically allocate and manage resources for applications and infrastructure. Challenges and Best Practices in GitOps Common Challenges in GitOps Implementation GitOps is not without its challenges, as it requires a shift in mindset, culture, and processes for software development and deployment. Some of the common challenges that developers may face when implementing GitOps are: Synchronization Issues: One of the main challenges in GitOps is ensuring that the system state is always synchronized with the code and configuration files in Git. This requires a reliable and robust automation tool that can detect and correct any discrepancies between the actual and desired state. It also requires clear communication and coordination among developers and other stakeholders to avoid conflicts or inconsistencies. Secrets Management: Another challenge in GitOps is managing secrets and sensitive information, such as passwords, keys, tokens, etc., that are required for accessing or deploying the system. Storing secrets in plain text in Git is not secure, as it exposes them to unauthorized access or leakage. Therefore, developers need to use encryption or other methods to protect secrets and use tools that can securely inject secrets into the system at runtime. Best Practices for GitOps Success To overcome the challenges and achieve success with GitOps, developers need to follow some best practices that can help them implement GitOps effectively and efficiently. Some of these best practices are: One of the best practices for GitOps is to securely manage access control and permissions for both code and configuration files in Git. Developers need to use authentication and authorization mechanisms to ensure that only authorized users can access or modify the code and configuration files. Developers also need to use encryption and signing methods to ensure that the code and configuration files are not tampered with or corrupted. Securely Managing Access Control and Permissions: I Another best practice for GitOps is to implement effective monitoring and alerting systems for both code and configuration files in Git. Developers need to use tools that can monitor the status and health of the code and configuration files, as well as the system state. Developers also need to use tools that can alert them of any changes, errors, or issues that occur in the code and configuration files, or the system state. mplementing Effective Monitoring and Alerting: GitOps is not just a theoretical concept, but a practical approach that has been adopted by many organizations across different industries. Some of these organizations are: is one of the world’s leading streaming platforms, serving over 200 million subscribers across 190 countries. Netflix uses GitOps to manage its complex and dynamic infrastructure and application state on AWS. Netflix uses tools such as Spinnaker, Terraform, and Chaos Monkey to automate its deployment and synchronization processes, as well as its resilience testing. Netflix also uses tools such as Atlas, Edda, and Vizceral to monitor and visualize its system state. Netflix: Netflix : Weaveworks is one of the pioneers and advocates of GitOps, providing a platform and tools that enable developers to implement GitOps workflows on Kubernetes. Weaveworks uses GitOps internally to manage its own infrastructure and application state on Kubernetes. Weaveworks uses tools such as Flux, Helm, Prometheus, and Grafana to automate its deployment and synchronization processes, as well as its monitoring and alerting systems. Weaveworks Intuit is one of the world’s leading financial software companies, providing products and services such as QuickBooks, TurboTax, and Mint. Intuit uses GitOps to manage its infrastructure and application state on Kubernetes. Intuit uses tools such as ArgoCD, Kustomize, and Istio to automate its deployment and synchronization processes, as well as its service mesh. Intuit also uses tools such as Wavefront, PagerDuty, and Slack to monitor and alert its system state. Intuit: GitOps is not a final destination, but a journey that continues to evolve and improve over time. As the software development and deployment landscape changes and advances, so does GitOps. Conclusion GitOps is a powerful and promising approach that can transform your software development and deployment processes. By using Git as the source of truth for both code and configuration files, you can achieve faster, smarter, and safer software development and deployment. It is more than just a technical approach, but a cultural and organizational one. It is a way of thinking and doing software development and deployment. However, remember that GitOps is not a one-size-fits-all solution, but a flexible and adaptable one. You need to continuously learn and adapt to the changing requirements and challenges of software development and deployment. You also need to experiment and innovate with new tools and technologies that can enhance your GitOps workflows. We believe that GitOps is the future of software development and deployment. We invite you to join us in this journey and embrace the transformative power of GitOps. Reference What is a GitOps workflow? | GitLab Guide To GitOps (weave.works) How to Deploy to Kubernetes using Argo CD and GitOps | DigitalOcean IBM Developer GitOps Workflow - Introduction to GitOps - Flux Based Course (cloudacademy.com) What is GitOps? (redhat.com) Guide To GitOps (weave.works)

A Deep Dive Into the GitOps World

322 Stories To Learn About Learning

Building a CI/CD Pipeline with AWS, K8S, Docker, Ansible, Git, Github, Apache Maven, and Jenkins

AWS, K8S, Docker, Ansible, Git, Github, Apache Maven এবং Jenkins এর সাথে একটি CI/CD পাইপলাইন তৈরি করা

Aufbau einer CI/CD-Pipeline mit AWS, K8S, Docker, Ansible, Git, Github, Apache Maven und Jenkins

Creación de una canalización de CI/CD con AWS, K8S, Docker, Ansible, Git, Github, Apache Maven y Jenkins

Création d'un pipeline CI/CD avec AWS, K8S, Docker, Ansible, Git, Github, Apache Maven et Jenkins

AWS, K8S, Docker, Ansible, Git, Github, Apache Maven और जेनकिंस के साथ CI/CD पाइपलाइन का निर्माण

AWS、K8S、Docker、Ansible、Git、Github、Apache Maven、Jenkins を使用した CI/CD パイプラインの構築

AWS, K8S, Docker, Ansible, Git, Github, Apache Maven 및 Jenkins를 사용하여 CI/CD 파이프라인 구축

Construindo um pipeline de CI/CD com AWS, K8S, Docker, Ansible, Git, Github, Apache Maven e Jenkins

Создание конвейера CI/CD с помощью AWS, K8S, Docker, Ansible, Git, Github, Apache Maven и Jenkins

AWS, K8S, Docker, Ansible, Git, Github, Apache Maven ve Jenkins ile CI/CD İşlem Hattı Oluşturma

Xây dựng quy trình CI/CD với AWS, K8S, Docker, Ansible, Git, Github, Apache Maven và Jenkins

使用 AWS、K8S、Docker、Ansible、Git、Github、Apache Maven 和 Jenkins 构建 CI/CD 管道

87 Stories To Learn About Tools

Hi Hackernoon! Tech experts warmly welcomed my . Even a top-level expert from Weaveworks contacted me after reading it. And I was greatly surprised. After some warm words, we agreed that they would wait for the second part of my article. Well, the time has come. previous article on GitOps Let’s dive in! Content Overview The Multiple Environments Problem Values From Secrets Issue CI Ops vs GitOps So, Who Really Needs GitOps? Conclusion The Multiple Environments Problem We've already dealt with Flux workflow in the previous part. Now let's move on. Here is our Helm release. Suppose we rolled out changes to the development environment, tested everything, and wanted to roll out to the stage. How can we do that? We have GitOps, so obviously, we want to maximize the Git approach (merge code from dev to stage). But in this Helm release configuration, we can't do that because (for example) we have a different base in the stage environment (lines 20...23). What does GitOps say about managing environments? , GitOps says "Why don't you just have one environment, and you won't have to do anything stupid". I, for one, am not sure how many teams have only one environment. GitOps ideologues don't seem to be sure about that either. So they say, "Well, if you really need multistage, do it somewhere outside of the GitOps scope. Within your system, for example." CI/CD Values From So we think, review the options, and find a few. The first one is (lines 15, 16, 17). valuesFiles We can describe separate values files for individual environments and plug them into HelmRelease. But this scheme doesn't fit well to git merge either. Next, we try valuesFrom. Our environment namespace may contain one or more config maps and/or secrets (lines 22...25). From them, Helm-controller can take Values and customize our Helm chart. And I like that. I even make a separate Helm chart where I configure all microservices, link it to the application charts with the dependsOn key (line 16), and proudly call it a single point of platform configuration. And then, the following happens: I have several Kubernetes clusters. My colleagues come and say: "Let's write another config map, and there will be cluster-specific Values (lines 26...28). For example, cluster name, Ingress suffix, region, and access rights. And we will create this config map with , together with the cluster. Terraform When this happens, I need to take a weightier argument and consistently explain the depth of their downfall to my colleagues. Because in this particular case, our application configuration will depend on 4 repositories (application chart repository, platform setup repository, flux infrastructure repository, terraform file repository) and two deployment systems (Flux and terraform). This is where I can give you some advice. If you want to work with , you should have only one value in your variables and config map. It must be the name of the environment you are deploying to. Everything else should be done inside the Helm chart using templating mechanisms. valuesFrom Secrets Issue Next point. We have a Helm release with a line that should not be there - the database password (line 23). What does GitOps tell us about managing secrets? , It says Never store your passwords in clear text in a Git repository. What can we say? Thanks, Cap! The next thing GitOps offers us is: Let's generate your passwords randomly when you create your environment. And you will make all objects that require password protection based on those passwords. That way, the password will never leave the environment, everything will be secured, and everyone will be happy. Nice, sure. But how many people, for example, have their SQL database in Kubernetes? Or is a cross-plane installed there? As a rule, we don't create heavy, persistent services inside or from inside a Kubernetes cluster. : Read on for the following alternative approaches Let's use tools like Mozilla SOPS or Bitnami Sealed Secrets when we do public and private keys. The private key is written to the cluster. With the public key, we encrypt our passwords and commit to a Git repository. At the time of deployment, some tool inside Kubernetes decrypts this and gives it to applications. And a specially trained Ops team member will write this key to the cluster. Let him carry a printout of it in his jacket pocket and then carefully enter it by hand. OK, that seems like the best option. Now let's think about it. In , old but still relevant, there is, for example, the 31st question: "Can a user's account be disabled on all systems in 1 hour?". Your Ops guy quit. And if your Kubernetes clusters are not under a single sign-on system, then you must run across all clusters and delete his account from everywhere. The Limoncelli Test: 32 Questions for Your Sysadmin Team The next question is, "Can you change all privileged (root) passwords in 1 hour?". That secret key that your Ops guy compromised by getting fired is that root password. And to change it, you have to do certain things: Extract the private key from the cluster Decrypt what we have in the Git repository with it Make new keys Write the private key to the cluster Encrypt the passwords Commit them to git Run it through the release cycle and deprecate it And you have to do it in a particular order because we have an asynchronous pool model. Any careless rollout can turn your environment into a pumpkin. And rollback won't help here. A couple of words for fans of external secret stores like Hashicorp Vault. I haven't found a consensus on them, but from a common-sense point of view, using external secret stores makes Git an even more non-unique source of truth and thus goes against the GitOps approach. CI Ops vs GitOps As part of the GitOps concept's PR, WaveWorks has often referred to classic CI Ops as an outdated approach and anti-pattern. Why? Because with GitOps, we can separate the stage of the deployment, protect it, and manage it more clearly. Within CI Ops, the deployment stage becomes a part of the monolithic pipeline. And because of this, several disadvantages arise. We will now analyze this fact. But! To get a fair comparison, we must choose a "fighter" from the CI-systems side. Let's take the popular and familiar to many people Gitlab, which implements the concept of pipelines as code. Your CI code is in the same repository as your application and is actually part of it. This allows for greater CI/CD transparency, self-documenting pipelines, and knowledge transfer within the team... CI Ops vs GitOps: Security The first problem is poor security. It's like we have a CI system within CI Ops, and developers have full access to Kubernetes through it, and it's non-secure. And within GitOps, we have an asynchronous executor pool that we don't have direct access to, greatly increasing the system's security. Let's put Flux in and see what the actual security is. By default, Flux gives cluster administrator rights to all of its controllers. Obviously, it's not very secure. Let's try to improve this situation. I have found an for Flux multiuser use. We create a tenant and see that namespace and ServiceAccount are generated. article about multi-tenancy Flux controllers get administrator rights to this namespace, not all of them, just Kustomize controllers. Earlier, using to limit the Flux Helm controller was recommended. Now, they offer some strange mechanism with patching of controller's deployments so that it starts to pick up the account service from the namespace where it is deployed. Kyverno But let's just think about it, abstracting from the toolkit. We have two schemes: the synchronous push and asynchronous pull models. Which one is safer, assuming they both have admin rights? That's a rhetorical question because admin rights are admin rights. But even if we restrict infrastructure rights, these schemes have roughly the same access to infrastructure, passwords, and persistent data. They can harm. If you look at the problem a bit from the outside, you can see that in both cases, you have a Git repository. And it contains both application code and infrastructure code. And probably the only way to stop all the abuse here is to control what gets into your Git. Gitlab has a mechanism for (though only in the paid version for now, but as we know, Gitlab's paid features are quickly moving to the community version). Merge request approvals This mechanism prevents you from merging your changes into the working branch of your repository until the right number of peers have reviewed your changes and confirmed that they are OK. This rather simple mechanism increases security many times over—much more than the difference between pull and push models. CI Ops vs GitOps: Rollback Procedure Another disadvantage that GitOps ideologues cite is the difficulty of rolling back CI. Their guides say that the CI system is not meant to be the only source of truth. This means that when working with the CI system, we cannot determine what we have clustered and what we will roll back to. But with GitOps, we have an infrastructure repository synchronizing with the infrastructure, so we do a git revert. And everything is fine. Let's think. GitOps only covers the deployment stage. But since git cannot be the only source of truth, we should think about rollback at the stage of building our artifacts. Besides, Flux has everything in place to make rollbacks difficult or impossible. For example, we can awfully tag a docker image (line 19). Also, we can use the Helm chart version range (line 11). And if we use Helm chart from the application repository, we may need to roll back the application repository to roll back our environment. In addition, don't forget about possible dependencies in a separate repository, which we may also need to roll back. valuesFrom Flux also has recommendations for organizing infrastructure repositories. For example, Monorepo. It is when we have the infrastructure for all our environments in one branch in different folders. Or, for example, Repo per team. It is when each team has its own folder in a shared infrastructure repository. This is where you must be very careful when rolling back so you don't delete extra. As for CI Ops, Gitlab may be the only source of truth. Gitlab has a so-called environment mechanism, where we specify the name of an environment in a Pipeline Deploy step and see its state in the environment tracking. It immediately becomes transparent which pipelines, from which branch, and from which commits are got. We also have a visual display of the pipelines, where we can see what rolled in there and whether those stages passed. We can roll back to the right stage. It is decisively impossible when looking at the commit history in an infrastructure repository in the GitOps concept. Alternatively, we can , thus making the pipeline atomic. And this would really be the only source of truth. save the docker image as an artifact in the pipeline As a result, to rollback within the implementation described above, you would just need to restart the Rollout Job in the Pipeline History in the Gitlab visual interface. CI Ops vs GitOps: The Problem of Multiple Clusters According to the GitOps concept developers, it is difficult to propagate changes to multiple clusters in CI Ops. Allegedly, in a CI system, we need to script, script, and script. But in the beautiful GitOps, we just put Flux in a new cluster and give it an infrastructure repository. One moment and our application is already in the new cluster. Let's start analyzing. Here is our custom resource Helm-release in the infrastructure repository. I have 2 questions when looking at this Helm release: Where did this application come from? Where is its repository? Where are the build logs and test run results? From which branch and from which commit is it deployed? Where is the application deployed to? How many fluxes in which clusters look at this infrastructure repository and pull configuration from there? "Out of GitOps scope" can be the answer to the first question. We just need to write a proper HelmRelease push script that will write all the information we need in the commit message. For the second question, Flux offers a with the creation of an infrastructure repository (flux-fleet) to manage Flux specifically. Before installing Flux in a cluster, we first commit all YAMLs describing the installation of flux controllers and related custom resources. Then we apply them to the target cluster. When we need to connect a git repository and Kustomize to a new infrastructure repository, we commit to that infrastructure repository again. bootstrap mechanism So we have the entire list of our clusters and all the accesses in one repository. And it works fine until some manual operations happen. For example, your Ops expert quits. Let's say there's a Kubernetes cluster somewhere where he put Flux and gave him manual bypass flux-fleet access to your infrastructure repositories. You may never know if you didn't do the access revocation and key regeneration. So I'd recommend writing down step zero on your checklist as creating a flux-fleet repository and bootstrap Flux. Otherwise, you will have secret knowledge that may one day be lost with the employee. What do we have in CI-ops? In Gitlab, we can declare reusable pieces of code (lines 1...9). We can inherit from these pieces of code (lines 13, 19, 25) and declare some additional variables (lines 15, 21, 27) when we make a new environment. In this case, it is the name of the cluster relative to which we will change the functioning of our scripts. In this way, we will deploy to a new environment and get its automatic tracking. So, Who Really Needs GitOps? There are a few cases where I think a GitOps approach can be justified: You don't have a CI/CD system, or it can't provide normal deployment for some reason. Improved security. GitOps can really add to the mix of measures to improve the security of your infrastructure. The lion's share of security issues are "out of GitOps scope" and should be addressed before or in conjunction with GitOps implementation. You have a large, multi-component platform that is tested and deployed. In this sense, a GitOps infrastructure repository is a good way to make a "stable" version of the platform and operate it as a single entity. You have processes where the pull model has advantages over the push model (e.g., canary deployments). So what is GitOps? For those who like precise formulas, GitOps = Continuous Delivery - Continuous Deployment. So it's not a CD concept. It's just (one more) way to organize automatic deployment. By and large, this methodology brought almost nothing new. The methodology authors took the ideas of Infrastructure as code (IaC), added a PULL-based approach with a reconciliation loop from SCM of the last generation (puppet, chef), and made it mandatory to keep the configuration in GIT. Indeed, a new and key concept of GitOps is the concept of Git as a single source of truth, but this concept is clearly mediocre and has some flaws. In fact, it's an attempt to take a deployment step from an application delivery pipeline and put it into Git. One of the key principles of DevOps is that there are no bad actors. There are bad processes. Hence the blameless culture and the understanding that you need to organize processes in such a way that the problem becomes impossible. But because of the limited concept of GitOps, it has too many gaps and potential problems, too many things to do "out of GitOps scope". All this leads to misunderstanding of how to organize processes properly and, consequently, to workarounds with all the consequences. To build a good GitOps process, a company needs high IT maturity because the authors of the concept left out too many important things. So, GitOps does not provide any significant advantage over a well-organized CI Ops process, although it significantly increases the final system complexity and labor costs. And in my opinion, many GitOps tools understand this by implementing GitOps in addition to image building, e.g., Gitkube and JenkinsX. Conclusion What else are we missing to make us happy? We obviously don't have enough notifications. We have an asynchronous pull model. In this model, we don't understand what exactly this pull executor is doing on its side if it doesn't explicitly notify us. Let's add the installation and configuration of the notification controller to our checklist and match it. So, to work with GitOps, we need to do: Flux-fleet repository Bootstrap flux in a Kubernetes cluster using the flux-fleet repository Script for building and pushing images in the Docker registry Infrastructure Git repository Account for CI system access to the infrastructure GIT repository Script to generate and push the HelmRelease file Helm Repository Account for CI system access to the Helm repository Script to build and publish the Helm chart Flux account for the infrastructure repository Flux account for Helm chartʼs repository Flux account for the application repository Flux account for cluster access for an engineer on the Ops team GIT repository for "single point of platform configuration" Flux account to access the GIT repository Slack/Teams channel for Notification Controller Most of these items are "out of GitOps scope". And it's not a methodology problem. But as an experienced engineer, I understand that I can't do GitOps without these items. What about CI Ops? As part of CI Ops on Gitlab, we don't need to do many extra actions to make GitOps work. We need to: Make the CI script to build and publish a docker image Add Helm release deploy to the script via Helm apply Create an account for the CI system to access Kubernetes With that, deprived of the illusion that it's secure, we'll be more responsible in setting up access to the CI system and the cluster. In addition, we get a built-in integrated docker registry, built-in environment handling, built-in secrets handling, built-in notification of the execution of the pipeline stages, and some integrations.

What Is GitOps And Why Is It (Almost) Useless? Part 2

On a new airline. A stewardess enters the passenger cabin: "You are on our new airline. In the nose of the aircraft, we have a cinema hall. In the tail part - a hall of slot machines. On the lower deck - a swimming pool. On the upper deck - a sauna. Now, gentlemen, fasten your seatbelts, and with all these unnecessary things, we're gonna try to take off. Hi, my name is Andrii. I've been working in the IT industry for most of my life. I'm very interested in the evolution of infrastructure configuration management engineering. For the last 8 years, I've been involved in . DevOps One of the fresh popular trends is the concept of , introduced in 2017 by Alexis Richardson, the CEO of Weaveworks. Weaveworks is a large adult company that, in 2020, raised over 36 million in investment to develop its GitOps. GitOps My previous article discussed a . Now, I'm going to try to talk about the non-obvious challenges that may await you when adopting this concept. In short, GitOps is not a "Silver Bullet". You'll likely end up reorganizing with many complicated workarounds. I've been down this road myself and want to show you the most frustrating problems you can't see when reading other articles about GitOps. cost-cutting success story on how we switched from Elastic Stack to Grafana Content Overview What GitOps is and why you (don’t) need it Snowflake Servers Issue GitOps - A Panacea for all Your Problems (or not) The Logic of Using Flux with Helm Custom Flux Resources Checklist for GitOps Violation of the Single Source of Truth Concept Small Conclusion What GitOps is and why you (don`t) need it Let’s dive right in! Stateless and Stateful The most promising concept of infrastructure construction today is immutable infrastructure. Its key idea is to divide infrastructure into 2 fundamentally different parts: Stateless and Stateful. The Stateless part of the infrastructure is immutable and idempotent. It does not accumulate state (does not store data) or change its operation depending on the accumulated state. Instances of the Stateless part may contain some basic artifacts, scripts, and assemblies. As a rule, I create them from base images in cloud/virtualized environments. They are fragile and ephemeral: I deliver new versions of applications by recreating instances from new base images. Persistent data is stored in the Stateful part. It can be realized by the classical scheme with dedicated servers or by some cloud mechanisms (DBaaS, object, or block storage). To make this “zoo” manageable and work correctly, we need collaboration between engineering and DevOps teams, as well as fully automated delivery pipelines. CI part Extreme programming is one of the agile development methodologies. It is distinguished by many feedback loops, which allow you to maintain synchronization with the client's needs. We implement the automation of delivery pipelines using CI/CD systems. The term CI (Continuous Integration) was offered by Grady Booch in 1994, and in 1997 Kent Beck and Ron Jeffries introduced it into the discipline of extreme programming. In CI, we need to integrate our changes as often as possible into the main working branch of our project. This requires, first, a more granular decomposition of tasks: small changes are more atomic and easier to track, understand, and integrate. Second, we can't just merge freshly written code. Before merging branches, we must ensure that nothing that worked before has been broken. To do this, the application should at least be built. It is also a good idea to cover the code with tests. And this is the task performed by CI systems, which have gone a long way in the development and, somewhere in the middle of this path, turned into CI/CD systems. CD part What is a CD? : Martin Fowler distinguishes 2 CD definitions Continuous Delivery. This is when, with the help of Continuous Integration practices and DevOps culture, you keep the main branch of your project constantly ready to be deployed to production. Continuous Deployment. It is Continuous Delivery where everything that goes into the main branch gets dumped into your cluster, into your production. Let’s go further. Snowflake servers issue Unfortunately, immutable infrastructure has several problems. The lion's share of them is inherited from the concept of Infrastructure as Code (IaC). First of all, it is configuration drift. This term was born in the Puppet Labs (authors of the well-known Puppet SCM) and states that not all changes on target systems are made with the help of system configuration management (SCM). Some are done manually, bypassing them. In the process of such multiple changes, configuration drift appears - the difference between the configuration described in SCM and the real state of affairs. This leads to an automation fear spiral. The more manual changes made, the more likely that running an SCM script will break unrecorded changes. The scarier it is to run it, the more likely new manual edits will be made. Eventually, this vicious positive feedback leads to the formation of snowflake servers, which have become so inconsistent that no one understands what's inside anymore. After manual edits, the node becomes as unique as each individual snowflake in a snowfall. This drift leaves the servers at higher levels within immutable infrastructure: now we can talk about GCP Project/AWS VPC/Kubernetes-cluster-snowflakes. This happens because the implementation of changes is not regulated on immutable infrastructure. Moreover, nobody knows how to do it properly. GitOps - a panacea for all your problems (or not) And then Weaveworks comes along and says, "Guys, we have what you need - GitOps". To promote GitOps, they brought in a heavyweight like Kelsey Hightower, who created the . During his PR, he heavily broadcasts the message, "Be a man, b...! Stop Scripting and Start Shipping." And he says a certain amount of marketing bullshit bingo. "Kubernetes the hard way" guide In my opinion, the most exciting benefits were: Improved consistency and standardization of deploys Improved security assurance Easier and faster recovery from errors Easier management of accesses and secrets Self-documenting deploys Knowledge distribution within the team And anyone trying to figure out what GitOps is comes across this textbook slide. Next, we find the GitOps principles, which resemble slightly augmented IaC principles: GitOps is declarative GitOps apps are versioned and immutable GitOps apps are pulled automatically GitOps apps are continuously reconciled Nevertheless, this is a spherical description in a vacuum, so we continue our research. We find the GitOps.tech website and on it several important clarifications. First, we learn that GitOps is an infrastructure-like code in Git with CD tooling that automatically applies this to the infrastructure. We must have at least 2 repositories within GitOps: Application repository. It describes the application source code and manifests that describe the deployment of that application. Infrastructure repository. It describes the infrastructure manifests and the deployment environment. Also, in the GitOps ideology, a pull-oriented approach is preferred over a push-oriented approach. This is somewhat contrary to the evolution of SCM systems from the heavyweight pull monsters Puppet and Chef to the lightweight push-based Ansible and Terraform. And if GitOps is primarily a toolkit story, then it makes sense to take the Flux-based concept from Weaveworks itself and deconstruct it. The authors of the idea must have made a reference implementation. Flux is now up to version 2 and architecturally consists of controllers that work within a cluster: Source controller Kustomize controller HELM controller Notification controller Image automation controllers Next, let`s discuss the work with Flux and Helm. The logic of using Flux with Helm I'm going to further describe the example of deploying an application using Helm package manager in Flux 2. Why? According to , HELM package manager was the most popular Packaging application, with a share of more than 50%. CNCF Survey 2021 Unfortunately, I couldn't find more up-to-date data, but I don't think anything much has changed since then. So, let's walk through the basic logic of how Flux 2 works with Helm. We have 2 repositories: application and infrastructure. We make a HELM chart and docker image from the application repository and add them into the Helm chart repository and docker registry, respectively. Next, we have a Kubernetes cluster running the flux controllers. To roll out our application, we prepare a YAML describing the custom resource (CR) HelmRelease and add it to the infrastructure repository. To help flux get it, we create a CR GitRepository in the Kubernetes cluster. The source controller sees it, goes to git, and downloads it. To deploy this YAML into a cluster, we describe a Customization resource. The Kustomize controller sees it, goes to the Source controller, gets the YAML, and deploys it to the cluster. The Helm controller sees that a CR HelmRelease has appeared in the cluster and goes to the Source controller to get the HELM chart described. For the Source controller to give the HELM controller the requested chart, we must create a HelmRepository in the CR cluster. Helm-controller gets a chart from Source-controller, creates a release, and deploys it to the cluster. Then Kubernetes creates the necessary pods, goes to the docker registry, and downloads the corresponding images. Accordingly, to roll out a new version of our application, we have to make a new image, a new HelmRelease file, and possibly a new HELM chart. Then we must put them into the appropriate repositories and wait for the Flux controllers to repeat the work in the chain described above. And, to end our work, we put a Notification controller somewhere that notifies us of what might have gone wrong. Custom Flux resources Now let's discuss the custom resources that Flux operates with. The first one is the Git repository. Here we can specify the address of the Git repository (line 14) and the branch it looks at (line 10). Thus, we only download a single branch, not the whole repository. But! Since we are responsible engineers and try to adhere to the Zero Trust concept, we lock access to the repository, create a secret with a key in the Kubernetes cluster and give it to Flux so that it can go there (line 12). Next is Kustomization. Here I want to draw your attention to the fact that the Kustomize controller from Flux and Kustomize from the authors of Kubernetes are 2 different things. I don't know why such disorienting naming was chosen, but it is important not to confuse them. Kustomization is a way to deploy YAML (any) from a Git repository to a cluster. Here we have to specify the source where we put it from (line 12 - the name of the CR GitRepository described above), the directory where we take the YAMLs from (line 8), and we can specify the target namespace where to deposit them (line 13). Next is the Helm release. Here we can specify the name and chart version (lines 10,11). Here you specify variable values so Helm can customize the release from environment to environment (lines 15-19). This is an extremely important and necessary feature, as your environments may differ significantly. You also specify the source to take the Helm chart (lines 12, 13, 14). In this case, it is the Helm repository. But! Since we are still responsible engineers, we also have close access to the Helm repository and give Flux a secret to get there (lines 7, 8). Checklist for GitOps So, let's do a little checklist to capture what we just went over. To start doing GitOps, we have to suddenly write a bunch of scripts (we do remember that Immutable infrastructure is all about fully automated delivery pipelines). So first of all, we have to create: Script to build and push images to the Docker registry Infrastructure Git repository Account for CI system access to the infrastructure GIT repository Script to generate and push the HelmRelease file Helm Repository Account for CI system access to the Helm repository Script to build and publish Helm chart` Flux account for infrastructure repository Flux account for the Helm chart repository Great, now you have a checklist for GitOps. Move on. Violation of the Single Source of Truth Concept Let's see what we get with our Helm release in general. It is quite evident that Git cannot be the only source of truth in this particular case. We have at least 2 resources, 2 artifacts outside of git, on which this Helm release depends: Helm chart (lines 8-14) Docker image (line 19) And we can complicate things even more and specify the range of Helm chart versions. In this case, Flux will monitor and set new Helm charts that appear within this range. In addition, the Source controller we have can use YAML as a source, including S3 bundles. From there, we can keep both YAML and Helm charts. In addition, we have Image automation controllers that can keep an eye on new images in the Docker registry and edit the infrastructure repository. But we don't want HELM Chart repo-Ops or Docker registry-Ops. We want to be as GitOps as possible. So we look at the documentation and correct the processes to deploy our Helm chart from the GIT repository (we choose the application repository to store it). This forces us to make another CR GitRepository for the application repository, an account for Flux to access it, and create a secret with keys. At the same time, we do not solve the problem of a complicated dependency on Docker image in any way. Small Conclusion I think that's enough for today. In the 2 part, I will tell you what problems this goodness has. I will discuss: Multiple environments problem Values from Secrets problem CI Ops vs GitOps Security Rollback procedure Multiple cluster problem Who really needs GitOps? I hope this article was useful for you!

What Is GitOps And Why Is It (Almost) Useless? Part 1

GitOps কি এবং কেন এটি (প্রায়) অকেজো? অংশ 1

Was ist GitOps und warum ist es (fast) nutzlos? Teil 1

Qué es GitOps y por qué es (casi) inútil: Parte 1

Qu'est-ce que GitOps et pourquoi est-il (presque) inutile ? Partie 1

GitOps क्या है और यह (लगभग) बेकार क्यों है: भाग 1

GitOps とは何か、そしてそれが (ほとんど) 役に立たない理由: パート 1

GitOps란 무엇이며 왜 (거의) 쓸모가 없나요? 1 부

O que é GitOps e por que é (quase) inútil: Parte 1

GitOps Nedir ve Neden (Neredeyse) Yararsızdır? Bölüm 1

GitOps là gì và tại sao nó (gần như) vô dụng: Phần 1

什么是 GitOps 以及为什么它（几乎）没用：第 1 部分

338 Stories To Learn About Github

139 Stories To Learn About Cicd

A hot take 🔥🔥 from a kind place. Before I start throwing sparks around I want to make clear that I think there’s lots of benefits to capturing in git. Static definitions, recipes and specs for how we make our software are useful in all kinds of ways. 🌈 everything as code However, those definitions to understand our dynamic environment and that’s my essential problem with GitOps. Lots of claims are made for GitOps - it offers better security, historical records, and a solution to drift and reconciliation. I find myself wondering whether any of this is really true and in this article I’ll explain why. don’t help us GitOps makes me think of the old Hans Christian Andersen tale, about what’s real and what’s imagined. The emperor declares he’s wearing clothes, but what if he’s actually not wearing anything at all? What is GitOps? Before we dig in, let’s set a baseline for what based on we describe as GitOps weavework’s four principles The entire system is described declaratively. The canonical desired system state is versioned in Git. Approved changes that can be automatically applied to the system. Software agents to ensure correctness and alert on divergence. Just like the , these four principles are pretty easy to accept. But, as with agile, turning theory into practice is the interesting part. agile manifesto What does GitOps look like in practice? Gitops strongly centers on the idea of software agents continuously running to converge system state with desired state. So, how do we reconcile these states using a typical GitOps approach? We install an operator (or agent) into our cluster which “pulls” (more on that later) the desired state from a git config repo, makes decisions, and adjusts workloads accordingly. This is offered as an alternative to a standard DevOps pipeline which “pushes” change to the cluster: Ok, so we’ve outlined the theory and described the basic practice. Now for the upsides for GitOps. How do they materialize when we start to implement? Extra security with GitOps? 🧐 First up - added . What’s the benefit of taking the “pull-based” approach as opposed to simply pushing a change to our cluster? The main advantage is that with GitOps your CI server doesn’t have production access, so we can say that this improves our security. security Traditional DevOps GitOps Infrastructure Imperative Declarative Declarative Desired State Untraceable Versioned Versioned Change Approvals Tickets Pull Requests Pull Requests Deployments Manual CI Event + GitOps Operator Security Manual Secrets in CI + Secretes in infra However, is there really any additional security in this setup? If the CI system can update configurations, how does GitOps prevent rogue workloads from being deployed by a malicious actor with access to CI? 🤔 Versioning and environment history Another major selling point for GitOps is the versioned history for the environment. That is kinda true, but you also get this with plain old DevOps assuming your pipeline and deployment information is in the source repo. This history is useful, but it isn’t a true record for how environments have actually changed (more on this later). Rollback Is rollback simpler with GitOps? I’m of the opinion that you’re better off with regular old DevOps by just reverting the commit. The benefit here is that it makes rollback a standard developer workflow and versioned with the source repository. Something doesn’t work? Simply git revert Disaster recovery What happens when your whole cluster goes down? What happens when you want to bring up a new cluster? Those are fair questions. But most teams aren’t rolling out blue/green clusters. Most companies have a static cluster/clusters. Most disaster recovery wouldn’t be hampered with the need to run deployment pipelines, and I think this should be scripted without the need for GitOps. So, yeah, I’m skeptical about the benefits. But I have more reservations when we start to look at the trade-offs we have to make when we implement GitOps. Let’s take a look at those. The challenges with GitOps The first big with GitOps is the effect it has on our pipelines. Splitting the deployment away from the earlier stages of the pipelines causes them to become distributed. From a value stream perspective this makes it hard to understand the overall path from commit to production. It disconnects earlier stages of qualification from later ones. challenge This matters because it removes the developer feedback from the value stream. In this setup, if a deployment fails, where does the feedback come from? How do developers get information on the deployment process? How can they enhance the deployment process with their own notifications? How can they improve the deployment process? The second side effect is that separating these stages into two toolsets increases the gap between development and operations. Typically the GitOps tooling is run and managed by a central platform team. Often the CI system is in the domain of the team. As Marshal McLuhan said, “We shape our tools and thereafter our tools shape us”. The gap is widened even further by using a separate config repo to store the desired states: It is common to have git repositories centered around individual microservices with a separate common repo for describing the desired state of environments. One is code and developer centric, one is operations centric. Also, it is not unusual to have to write glue pipeline scripts to update the config repo. Revisiting push vs. pull The major innovation in GitOps seems to be to move the operations to a . This seems like a big change, but on closer inspection I don’t think it’s actually true. [Thanks to my good friend Henrik Hoegh for spelling this out to me] pull-based model Typically, a GitOps operator reads a config from a git repo, applies zero-to-many transformations on it, and then into the kubernetes API server. Which is exactly what your deployment tool does in the push based model! With GitOps we distribute our pipeline over two asynchronous tools, using a git repository as a semaphore, but . pushes it with both approaches we changes into our cluster push It’s great for Drift and Reconciliation though, right? Another big headline advantage of GitOps is the reconciliation loop - the automated repairing of any or manual change. Any undocumented changes are erased and the environment is reconciled with the git definition. drift At face value, this seems like a massive bonus. However, I take a different view on this too.  Before we jump to reconciling undocumented changes we need to ask . Maybe we don’t want them reconciled? There could be a very good reason to do a manual change, and we might not want the environment to be automatically repaired. why they have happened in the first place Another reason might be sabotage, in which case we definitely want humans in the loop to investigate and manage the situation. In either case, to take place, not just a slack message from a reconciliation loop that disappears into the ether. configuration drift should cause a proper incident management process And on the technical side, I feel that Kubernetes already has a reconciliation loop. You describe your deployments and configuration declaratively, and it is Kubernetes’ job to make that true.  Layering reconciliation loops feels like adding unnecessary complexity. The map git repo is not the territory We like to think the git config repository is equivalent to how things change, but in reality there is a gap between these static definitions and what is actually happening in the dynamic DevOps automation. All this talk about GitOps providing a “Single Source of Truth” is simply not true. If I want to find out what was actually running on Thursday night there is no simple way to get there. The GitOps configuration provides no insight into manual changes, scaling events, failed reconciliation and many other edge cases. These types of events cause incidents but GitOps provides no situational awareness when they happen. When an incident occurs what we really need is to understand how things have changed. A big problem with modern GitOps is developers and ops teams having little or no true record of the actual changes that occur. We need to be clear that desired states are not actual states. actually A static view of change is beneficial but limited I began by saying that I’m all for putting the recipes, definitions, and specs for our desired DevOps in version control. It offers us all kinds of benefits. Let’s remind ourselves of what they are: : enables sharing, reviewing and auditing in a familiar technology Better transparency : enables branching/pull-request based approaches to integrate change Code tools and workflows : allows you to add linters, checkers and static analysis in the automation processes, and enforces consistency of changes Better Quality : helps minimize configuration drift Immutability : can help reduce “configuration sprawl”: the configuration of processes spread over multiple unconnected systems Centralization So far so good - but every static definition has a dynamic execution. There are real events taking place, asynchronously and automatically, with results we need to record and understand. Static Definition Dynamic Execution Build script Compilation/Packaging Test suite Test runs Deployment file Deployments Docker file Docker image builds Infrastructure model Infrastructure changes The dynamic world is, quite literally, where the action is. Working back from a GitOps definition to events, changes, ordering, and dependencies is not easy for developers. If you’re still wondering why this matters, the tells us that “70% of outages are due to changes in a live system.” So, when things go wrong, the dynamic world should be the first place we look for answers. google SRE book Conclusions - Who’s GitOps? Much like the agile manifesto, the loose definition of GitOps means that it can and will be applied in all kinds of different ways. A broad church approach is the perfect nerd-snipe. Is GitOps? Maybe? I dunno! Terraform And like agile, everyone experiences FOMO. What if this is the next big thing? Do we jump on the bandwagon for fear of being left behind? With agile we need to ask “who’s agile?” Maybe we need to ask “who’s GitOps?” As ever, what we really need to ask ourselves is - who are we serving with these tools and what problems are we trying to solve? At the end of The Emperor’s New Clothes the people around the emperor continue to praise his new outfit, even as they come to realize he’s completely naked. It’s an embarrassing situation that any of us can get into when . Like the kid in the crowd who shouts “But the Emperor has nothing at all on!” sometimes it’s good to uncomplicate everything by just saying what you see. we go all in on something and want it to be true

Is GitOps the Emperor With No Clothes?

Stories, News & Mentions about WEAVEWORKS

Weaveworks’s Stories on HackerNoon