Bluelight Consulting

bluelightconsulting

As a software professional handling Infrastructure as Code (IaC), chances are you work a lot with . When helping new clients to use IaC, it is common to simplify things, but managing a Terraform state file is the first challenge you face. Essentially, Terraform state contains sensitive information which shouldn’t be stored by source control but, at the same time,  won't scale if you have multiple users working on the same Terraform state.  The answer to that? Backends. Terraform It is important to note that you could store that state file on an S3 bucket and use DynamoDB to manage the locking state. However, this approach will force you to create additional resources which makes it a complicated option, especially if the client is using GitLab. GitLab recently lowered the entry barrier to integrating Terraform by providing a way to store and manage Terraform state, as well as an easy way to set up a CI around it. In this blog post, we will explain what a Terraform state file is, how to migrate it to GitLab, and setting up a CI Pipeline for it. You can visit our repository . here Table of Contents What is Terraform state? How to get GitLab to manage Terraform state How to get GitLab to run your IaC through a CI Pipeline Bonus Tip:  Infracost Conclusion What is Terraform State? Terraform records any information about the infrastructure defined in your code via a state file. Written in JSON, it essentially records a mapping from the Terraform code to the real resources created. Below is an example of what a terraform.tfstate would look like. Primarily, every time you run Terraform it will fetch the latest status for its EC2 Instance and compare it with your Terraform configuration to determine what changes need to be applied. {
    "version": 4,
    "terraform_version": "0.12.0",
    "serial": 1,
    "lineage": "1f2087f9-4b3c-1b66-65db-8b78faafc6fb",
    "outputs": {},
    "resources": [
      {
        "mode": "managed",
        "type": "aws_instance",
        "name": "example",
        "provider": "provider.aws",
        "instances": [
          {
            "schema_version": 1,
            "attributes": {
              "ami": "ami-0c55b159cbfafe1f0",
              "availability_zone": "us-west-2a",
              "id": "i-00a123a0accffffff",
              "instance_state": "running",
              "instance_type": "t2.micro",
              "(...)": "(truncated)"
            }
          }
        ]
      }
    ]
  } By default, this terraform.tfstate is stored locally where you have your Terraform files, plan, and apply your changes. For a personal project where you are just running some tests, it's fine but not the recommended way, here's why: if you were hosting this state file on your local workstation and had to work with another engineer, things would get complicated. Both of you will have to make sure you are using the latest version of the state and you could run into race conditions if you run a Terraform plan or apply at the same time. Stored in a shared location: A generated state file can contain encryption keys and infrastructure passwords. However,  state files aren't encrypted by default, and storing sensitive information in plain text is a bad idea. Protect sensitive information: Most version control systems do not support any form of locking which prevents two team members from running Terraform apply simultaneously on the same state file. This is another reason why we will not see a state file managed by source control. Locking: ‍ How to Get GitLab to Manage Terraform State With Terraform being considered the standard in cloud infrastructure provisioning, it has been a year or so since GitLab began to offer a way to store and manage your Terraform state. For this reason, we wanted to share the migration process with you as we recently started using GitLab to manage our IaC. For this article, we presume that you are using a local state, and have your state managed with an AWS S3 Bucket or another backend solution. Firstly, you will need to change your backend.tf to use HTTP. terraform {
  backend "http" {}
} Next, you will need to set up 4 variables in your terminal: You can find this easily by navigating to your repo on the Project Overview page PROJECT_ID: __ __ ‍ Gitlab username that has access to the repo you're working on. TF_USERNAME: Access token generated from your GitLab user. TF_PASSWORD: URL of the remote state backend TF_ADDRESS: PROJECT_ID="28450092"
TF_USERNAME="florianpialoux"
TF_PASSWORD="123456789"
TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/aws-buckets" You may now run the migration command that will move your Terraform state from his previous location to GitLab with the following command: terraform init \
  -migrate-state \
  -backend-config=address=${TF_ADDRESS} \
  -backend-config=lock_address=${TF_ADDRESS}/lock \
  -backend-config=unlock_address=${TF_ADDRESS}/lock \
  -backend-config=username=${TF_USERNAME} \
  -backend-config=password=${TF_PASSWORD} \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5 You will need to provide a confirmation by a “yes” so that GitLab can start managing your state file. Here's an example from a local state to GitLab: __ __ Example s3 to GitLab __ __ Now you can navigate to from the GitLab interface and see your state: Infrastructure > Terraform __ __ I noticed for some of the state files I had from s3 will be blank even after using the migrate-state command ran previously, in this case, you can run this: terraform state pull > aws-buckets.json Copy and paste the content from the s3 state and run a push: terraform state push -lock=true aws-buckets.json GitLab supports versioning for your Terraform state file but viewing/restoring older versions through the WebUI will require you to be using a . If not, you will need to make a GraphQL API . GitLab Premium plan request ‍ How to Get GitLab to Run your IaC through a CI Pipeline GitLab provides that contains GitLab-Terraform, which is a thin wrapper script around the official Terraform binary. Alternatively, you could use the by Hashicorp. You can find more information about the GitLab Terraform Image . a docker image official docker image here Once the terraform apply job runs, you will be able to see when the state was used and with which pipeline. __ __ Learn more about what our looks like . Below are the variables that will need to be defined on the project level: gitlab-ci.yml here __ __ Bonus Tip: Infracost As you might have noticed, looking at our gitlab-ci.yaml we added which allows us to have more control over our cloud billing as it gives you a cost estimate whenever you define a new resource to your IaC. To learn more about Infracost, you can check out our in-depth about what it entails, and our step-by-step on how to integrate it with Terraform. Infracost article guide Conclusion Having your Terraform state and CI running on Gitlab is a great way to follow GitOps best practices. They both make a great combination to develop and deploy IaC. Since most of you might be already using GitLab for your repositories, it becomes much simpler to have your IaC under one roof and let GitLab manage your Terraform state by supporting encryption in transit and at rest, as well as versioning, locking, and unlocking the state. Also published . here

A Handy Guide to Migrating Terraform State to GitLab CI/CD

Una guía práctica para migrar Terraform State a GitLab CI/CD

Un guide pratique pour migrer Terraform State vers GitLab CI/CD

Terraform State को GitLab CI/CD में माइग्रेट करने के लिए एक हैंडी गाइड

Terraform の状態を GitLab CI/CD に移行するための便利なガイド

Um guia prático para migrar o estado do Terraform para o GitLab CI/CD

Hướng dẫn hữu ích để di chuyển trạng thái Terraform sang GitLab CI/CD

将 Terraform 状态迁移到 GitLab CI/CD 的便捷指南

Stories, News & Mentions about BLUELIGHTCONSULTING

Bluelightconsulting’s Stories on HackerNoon