Reducing the fear of introducing breaking changes in the cloud Infrastructure as Code is one of the cool things right now. Every DevOps-related conference in the past two years had a talk or two about the subject, and that’s a good thing. In the wake of the DevOps movement, emerged as one of the most respected companies in that space. Today I’m going to talk about one of their products: . HashiCorp Terraform What is Terraform? Terraform is a tool which allows to easily manage cloud resources in a declarative way. Using a simple Programming Language, it lets you define pretty much the shape of a cloud infrastructure including VPCs, Subnets, Compute Instances, Load Balancers, DNS Records and so on. It works with every major cloud provider, but it’s not cloud-agnostic. That means you can create for example a Load Balancer in AWS or Google Cloud, but the code will be slightly different for each of them. What is Blue/Green deployment? Blue/Green deployment is a DevOps practice that aims to reduce downtime on updates by creating a new copy of the desired component, while maintaining the current. Given that, you end with two versions of the system: One with the actual version ( ) and another with a newer one ( ). When the new version is up and running, you can seamlessly switch traffic to it. This is useful not only to reduce downtime, but also to improve rollback time when something bad happens. blue green Example 1 Example 2 Blue/Green Infrastructure While Blue/Green deployment is a technique more commonly used with application deployment, the reduced costs of the cloud, in conjunction with the tools we have right now, make possible to have two copies of an entire cloud infrastructure with little to no pain. It is important to note that doing Blue/Green deployment of an entire Cloud Infrastructure is not a silver bullet and certainly a bit too much if you are doing small changes (for example, adding a new EC2 Instance to your stack). But for major/breaking changes is a win and I personally recommend it. Terraform to the rescue! I’ll be using Amazon Web Services for this tutorial, but the code won’t vary too much with another provider. After finishing this, you will be able to create an infrastructure containing: A Virtual Private Cloud Three Subnets, each one in a different Availability Zone A Security Group Three EC2 Instances serving an NGINX Server on the Port 80 (each one in a different subnet) A Load Balancer pointing to those Instances Then, you will be able to: Make changes in the Infrastructure Create an entire new Infrastructure with that change Switch traffic through the new Infrastructure Destroy the Old Infrastructure Profit The full example can be seen on https://github.com/santiagopoli/terraform-examples/tree/master/blue-green To follow this tutorial, you need to have your AWS Credentials configured in your Environment, with at least the policy attached. EC2FullAccess Creating a VPC (Virtual Private Cloud) I know this is a Terraform tutorial, but a recommended practice is to have a manually created VPC. You can create VPCs with Terraform, but there are a lot of external services that rely on knowing your VPC ID beforehand, so it is better to not create a new one every time on every Blue/Green deployment. Also, you may have security groups that are created externally by another team in your organization. For that matter, we will be creating a VPC using the AWS Console. You can also create a VPC with the command line by doing: (change the CIDR block to anything you like) > aws ec2 create-vpc --cidr-block 10.0.0.0/16 {"Vpc": {"VpcId": " ","InstanceTenancy": "default","Tags": [],"CidrBlockAssociations": [{"AssociationId": "vpc-cidr-assoc-6e42b505","CidrBlock": "10.0.0.0/16","CidrBlockState": {"State": "associated"}}],"Ipv6CidrBlockAssociationSet": [],"State": "pending","DhcpOptionsId": "dopt-38f7a057","CidrBlock": "10.0.0.0/16","IsDefault": false}} vpc-ff7bbf86 Note your , as you will need it in a second. VpcId Installing Terraform You can download Terraform by either going to or by using any Package Manager (brew, apt) this link Creating the project Create a new folder in your workspace and name it Then, initialize a GIT repository, add a simple .gitignore that ignores the .terraform folder and open the folder with your favorite text editor. In my case, I’ll be using . terraform_blue_green. Visual Studio Code > mkdir terraform_blue_green> cd terraform_blue_green> git init> echo .terraform >> .gitignore> code . Initializing the Terraform State Terraform stores the state of the infrastructure in a JSON File. It is recommended (and required for this tutorial) to store that file on an external backend like Amazon S3. As I’m using AWS for this Tutorial, I’ll stick to S3, but . Terraform supports the equivalent in each provider First of all, you need to create the S3 bucket in which the state will reside. You can do this either by going to the S3 Console or by doing: > aws s3api create-bucket --bucket terraform-bluegreen Then, create a file named inside the project folder, with this content. bootstrap.tf In this file we have defined The of the Infrastructure version The we will be using (in this case, AWS) Cloud provider The in which the state will be saved (in this case, S3), and the configuration attached to it. Backend With that file in place, run this command in your project folder: > terraform init Initializing the backend... Successfully configured the backend "s3"! Terraform will automaticallyuse this backend unless the backend configuration changes. Initializing provider plugins...- Checking for available provider plugins on ...- Downloading plugin for provider "aws" (1.11.0)... https://releases.hashicorp.com Terraform has been successfully initialized! Using existing resources in Terraform As we’ll need the ID of the previously created VPC to do anything in our infrastructure, we will be storing it in a variable. To do this, create a file named with this content: vpc.tf Creating our first resource: Subnets To do anything useful, we first need subnets. We will create three of them, each in a different Availability Zones. Create a file named with this content: subnets.tf, In this file we created three subnets specifying: The number of Subnets we want to create Count: In this case we are using the function which function takes a list and an index and returns the element, even if the index is greater than the number of elements. This is useful to assign a different availability zone to each subnet. Availability zone: element() VPC ID This is probably the most confusing part. We interpolated the previously defined variable into the CIDR block. This will help in the future, when creating the second version. You may change the CIDR Block with the one you defined in the VPC. CIDR Block: infrastructure_version Assign a by default to any Network Interface assigned to this subnet Public IP We’ve appended the Infrastructure version into it Name: With the file in place, first do: > terraform plan + aws_subnet.terraform-blue-green[0]... + aws_subnet.terraform-blue-green[1]... + aws_subnet.terraform-blue-green[2]... Plan: 3 to add, 0 to change, 0 to destroy. The does a dry run and tells you what changes will be done. It is important to plan before doing anything, as you can spot errors. In this case, the plan tells us that it will add three subnets, and that’s what we wanted. plan command So now we can run this: > terraform apply Do you want to perform these actions?Terraform will perform the actions described above.Only 'yes' will be accepted to approve. Enter a value: yes aws_subnet.terraform-blue-green[0]: Creating......aws_subnet.terraform-blue-green[1]: Creating......aws_subnet.terraform-blue-green[2]: Creating......aws_subnet.terraform-blue-green[0]: Creation complete after 5saws_subnet.terraform-blue-green[1]: Creation complete after 5saws_subnet.terraform-blue-green[2]: Creation complete after 5s Apply complete! Resources: 3 added, 0 changed, 0 destroyed. Now, you can go to the AWS Console (under the section) and your subnets should appear VPC/Subnets Creating a Security Group To be able to access our resources in the future, we need to create a Security Group in our VPC. For the sake of simplicity, we will be creating a Security Group that enables all inbound traffic from everywhere. Create a file named in your project, with this content: security_groups.tf In this file, we’ve created a security group for our VPC (using its VPC ID) and two Rules: One for Inbound traffic and one for Outbound traffic. The most important parts are: The port range the rule applies for. In this case, we target all possible port ranges From/To Port: You can use either HTTP, TCP or “-1”, which applies for both TCP and HTTP Protocol: A list of CIDR blocks that are enabled by the rule. In our case, we enabled all ipv4 traffic. CIDR Blocks: With the file in place, run a and . After that, we should be able to see our security group in the AWS Console (under ). terraform plan terraform apply EC2/Security Groups Creating an SSH Key To be able to access an AWS Instance later in the future, we need to assign an SSH Key to it. First, create a Key Pair by using ssh-keygen: > mkdir keypairs > ssh-keygen -f keypairs/keypair -P "" Generating public/private rsa key pair.Your identification has been saved in keypairs/keypair.Your public key has been saved in keypairs/keypair.pub. Given this is a tutorial, don’t bother moving the Private Key to a secure place (but you should definitely do it). Then, create a file named in the root folder of the project. Give it this content: keypairs.tf Then do: > terraform plan + aws_key_pair.key_pair... Plan: 1 to add, 0 to change, 0 to destroy. > terraform apply Terraform will perform the following actions: + aws_key_pair.terraform-blue-green... Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions?Terraform will perform the actions described above.Only 'yes' will be accepted to approve. Enter a value: yes Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Now, the Key appears on the AWS Console (under ) EC2/Key Pairs Creating (at last!) EC2 Instances Create a file named , and paste the following: instances.tf Let’s explain a little bit about this file: We have created a resource of type , with this parameters: aws_instance The number of resources of this type. In this case, we will create 3 instances Count: The Amazon Image for the instances. In this case, we chose the Official AWS ECS Image, as we will run a docker container Inside of it. Keep in mind that this AMI only works in the US-WEST-2 region, so if you are in another region. AMI: check this link The instance type of the instances Instance Type: As explained above, the function takes a list and an index and returns the element, even if the index is greater than the number of elements. This is allows us to assign a different subnet id to each instance. Subnet Id: element() The name of the key pair. We chose the previously created one Key Name: This allows us to assign an initialization script to the instance. In our case, we are running an NGINX Docker Container and exposing it in the port 80. There are better ways to define User Data Scripts, but we’ll keep it simple for now. User Data: With the file in place, run: > terraform plan Terraform will perform the following actions: + aws_instance.terraform-blue-green[0]... + aws_instance.terraform-blue-green[1]...+ aws_instance.terraform-blue-green[2]... Plan: 3 to add, 0 to change, 0 to destroy. > terraform apply Terraform will perform the following actions: + aws_instance.terraform-blue-green[0]...+ aws_instance.terraform-blue-green[1]...+ aws_instance.terraform-blue-green[2]... Plan: 3 to add, 0 to change, 0 to destroy. Do you want to perform these actions?Terraform will perform the actions described above.Only 'yes' will be accepted to approve. Enter a value: yes .... Apply complete! Resources: 3 added, 0 changed, 0 destroyed. Outputs: instance_public_ips = [ip1,ip2,ip3] When the command finishes, you will be able to see the instances in your AWS Console As you see, every of them are on different availability zones. Accessing any of your instances (via their public IP) from a browser should display: Adding a Load Balancer Create a file named load_balancers.tf with this conent: In this file we’ve created a Load Balancer with : Self explanatory Name : The subnets the Load Balancer is Available In Subnets We’ve added the previously created Security Group to be able to access it Security Groups: We’ve added the previously created instances Instances: We’ve added a single listener that listens in the 80 port of the load balancer, and points to the port 80 of the instances Listeners: We’ve added a simple HTTP Healthcheck that targets the port 80 of the instance. Healthcheck: We’ve added an output that displays the Load Balancer Public DNS Outputs: With the file in place, do a and a When the execution ends, it should output the Load Balancer’s public dns. terraform plan terraform apply. > terraform apply .... Outputs:... load_balancer_dns = terraform-blue-green-v1-xxxxxx.us-west-2.elb.amazonaws.com Accessing the public DNS of the Load Balancer from a browser should display the NGINX page. Yes, I’ve used the same screenshot twice You should also be able to see it in the AWS Console (under ) EC2/Load Balancers (Optional) Assign A DNS Record to the Load Balancer V1 I’m not going to cover too much of this case, but what I’ve ended up doing in production is creating a DNS Record that points to a specific version of the Load Balancer. An example of this in terraform could be: Commit your changes Commit your changes so far > git add .> git commit -m "Version 1" Manually Pointing a DNS record to the Load Balancer . In some cases, it’s a good practice to have a minimal human interaction. In our case, we will assign a DNS record to the desired version of the infrastructure (via the load balancer). DevOps is not all about Automation To be able to perform this step you’ll need to have a registered domain and the corresponding Route 53 Hosted Zone. Enter the desired Hosted Zone and create an with an of the previously created (terraform-blue-green-v1…). A Record Alias load balancer This is the entry point of your system and what your clients will be accessing. Creating the Infrastructure V2 First, create a new branch in your repository (and I seriously recommend removing the .terraform folder): > git checkout -b v2> rm -rf .terraform Now, modify bootstrap.tf with this: As you see, you need to modify both the variable and the I’ll be nice if terraform allowed to interpolate the infrastructure_version variable in the key, but for now it’s not possible. There is an issue in Github though. infrastructure_version key of the S3 Bucket. Now, as you deleted the .terraform folder, you need to reinitialize the state: > terraform init Now modify your instances.tf with this content: (We’ve . You can chose whatever you like) changed the instance size from t2.micro to t2.medium Doing a will reveal that in fact, terraform will create all resources again. terraform plan ...Plan: 11 to add, 0 to change, 0 to destroy. > terraform plan After doing you should end with an entire new infrastructure, without changing the old one. terraform apply, Instances Subnets Security Groups Load Balancers Routing traffic through the new Infrastructure As we did previously with Version 1, point your DNS record to the new load balancer using an ALIAS. Deleting the old infrastructure When all traffic starts going to the new Load Balancer, it’s time to delete the Version 1 of the infrastructure. To do this, first commit all the changes in Version 2, and then checkout the old version again. Delete the .terraform folder and initialize the state again. > git add .> git commit -m "Version 2"> git checkout master> rm -rf .terraform> terraform init Then, simply do: > terraform destroy Terraform will perform the following actions: - aws_elb.terraform-blue-green - aws_instance.terraform-blue-green[0] - aws_instance.terraform-blue-green[1] - aws_instance.terraform-blue-green[2] - aws_key_pair.terraform-blue-green - aws_security_group.terraform-blue-green - aws_security_group_rule.terraform-blue-green-inbound - aws_security_group_rule.terraform-blue-green-outbound - aws_subnet.terraform-blue-green[0] - aws_subnet.terraform-blue-green[1] - aws_subnet.terraform-blue-green[2] Plan: 0 to add, 0 to change, 11 to destroy. Do you really want to destroy?Terraform will destroy all your managed infrastructure, as shown above.There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes ... Destroy complete! Resources: 11 destroyed. Now, if you go to the AWS console you should see only the V2 resources. For example, this is a screenshot of the Instances after destroying the Version 1: You can now merge the v2 branch into master. If you are doing this just for fun, please do a for v2 ;) terraform destroy Remember that you can see this full example on https://github.com/santiagopoli/terraform-examples/tree/master/blue-green A note of caution In this guide, we used a DNS Record to select which infrastructure version is the production one. While this works most of the time, there are some cases when some client-side libraries cache DNS Entries, so you should wait some time to get the traffic to drain from the old balancer. You can solve this by maintaining a manually-created load balancer and changing its instances. Conclusion Terraform provides a clean and declarative way of defining Infrastrucure as Code. Thanks to that, we can use it to perform things that seemed impossible a few years ago. I want to end this article by saying that this approach has a couple of downsides, and I will write an article in the future explaining how to achieve the same results by using Terraform Modules (those in fact provide better flexibility overall). Thanks for reading!
