Introduction to AWS With Terraform

Almost one month ago, I released Practical AWS, a training concerned With the actual use of AWS rather than with theory & ideas. You can watch the demo video, or visit the training website. demo video the training website More information on http://practicalaws.com http://practicalaws.com Even if the training is released, but since it will be lifetime updated with new contents for free, this blog post is a prototype of the new lesson that will be added to the training. This blog post is an introduction to managing an AWS infrastructure using Terraform. Downloading & Installing Terraform Start by downloading Terraform from the official download page . the official download page Choose your OS and CPU architecture and start the download. Terraform is a single binary that you should move to /usr/bin and make it executable. /usr/bin sudo mv terraform /usr/bin && sudo chmod +x sudo mv terraform /usr/bin && sudo chmod +x If you are using another OS, please refer to the documentation . the documentation Configuring AWS In order to follow the best practices, let’s create a user for Terraform. Go to your AWS console and create terraform_user user: terraform_user Give it the good rights. In my example, I need Terraform to be able to manage all of my AWS Cloud resources: Don’t forget to store the AWS access key id and secret access key: Copy them in your AWS credential file: .aws/credentials .aws/credentials You can also execute aws configure to add a new user. aws configure In both cases, your keys will be stored in the AWS credentials file: [terraform]aws_access_key_id = xxxxxxxxxxxxxxxxxxxaws_secret_access_key = xxx/xxxxxxxxxxxxx/xxxx [terraform]aws_access_key_id = xxxxxxxxxxxxxxxxxxxaws_secret_access_key = xxx/xxxxxxxxxxxxx/xxxx Terraform Hello World ! Go to your workspace and create a folder called terraform: mkdir terraform mkdir terraform Add these lines to main.tf : main.tf provider "aws" { region = "eu-west-3" shared_credentials_file = "/home/eon01/.aws/credentials" profile = "terraform"} provider "aws" { region = "eu-west-3" shared_credentials_file = "/home/eon01/.aws/credentials" profile = "terraform"} The above is the configuration for AWS, adapt the credential file path to your own configuration, the profile name, as well as the region. In my example, I am using the Paris region. In order to create our first AWS machine, let’s add these lines: resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } Our file main.tf will look like this: main.tf provider "aws" { region = "eu-west-3" shared_credentials_file = "/home/eon01/.aws/credentials" profile = "terraform"} resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } } provider "aws" { region = "eu-west-3" shared_credentials_file = "/home/eon01/.aws/credentials" profile = "terraform"} resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } } In the example above, I am creating a machine on the region “eu-west-3” using the profile “terraform”. My machine size is t1.micro and it is using the AMI ami-0e55e373 , which is a Ubuntu 17.04 image available for the region “eu-west-3”. t1.micro ami-0e55e373 Note : Ubuntu 17.04 image doesn’t have the same AMI id in two different regions. Note If you prefer using Ubuntu like in this example, you can visit cloud-images.ubuntu.com where you can find the id of the AMI you should use. cloud-images.ubuntu.com You can also use the CLI in order to describe AWS IAMs: e.g: Describing Windows AMIs that are backed by Amazon EBS. aws ec2 describe-images --filters "Name=platform,Values=windows" "Name=root-device-type,Values=ebs" e.g: Describing Ubuntu AMIs aws ec2 describe-images --filters "Name=name,Values=ubuntu*" e.g: Describing Windows AMIs that are backed by Amazon EBS. aws ec2 describe-images --filters "Name=platform,Values=windows" "Name=root-device-type,Values=ebs" e.g: Describing Windows AMIs that are backed by Amazon EBS. aws ec2 describe-images --filters "Name=platform,Values=windows" "Name=root-device-type,Values=ebs" e.g: Describing Ubuntu AMIs aws ec2 describe-images --filters "Name=name,Values=ubuntu*" e.g: Describing Ubuntu AMIs aws ec2 describe-images --filters "Name=name,Values=ubuntu*" Note: Check the AWS cheat sheet that comes with this training in order to get more examples. Note: After choosing the AMI, go into the folder where you created main.tf and initialize Terraform: terraform init terraform init You will see a similar output to this one: Initializing provider plugins...- Checking for available provider plugins on https://releases.hashicorp.com...- Downloading plugin for provider "aws" (1.13.0)... The following providers do not have any version constraints in configuration,so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breakingchanges, it is recommended to add version = "..." constraints to thecorresponding provider blocks in configuration, with the constraint stringssuggested below. * provider.aws: version = "~> 1.13" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to seeany changes that are required for your infrastructure. All Terraform commandsshould now work. If you ever set or change modules or backend configuration for Terraform,rerun this command to reinitialize your working directory. If you forget, othercommands will detect it and remind you to do so if necessary. Initializing provider plugins...- Checking for available provider plugins on https://releases.hashicorp.com...- Downloading plugin for provider "aws" (1.13.0)... The following providers do not have any version constraints in configuration,so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breakingchanges, it is recommended to add version = "..." constraints to thecorresponding provider blocks in configuration, with the constraint stringssuggested below. * provider.aws: version = "~> 1.13" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to seeany changes that are required for your infrastructure. All Terraform commandsshould now work. If you ever set or change modules or backend configuration for Terraform,rerun this command to reinitialize your working directory. If you forget, othercommands will detect it and remind you to do so if necessary. Now execute: terraform plan terraform plan This command will not create any resource on your AWS cloud. It lets you know what Terraform will do. You will see this output: + aws_instance.web id: ami: "ami-0e55e373" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: + aws_instance.web id: ami: "ami-0e55e373" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: Note that the (+) sign indicates that a resource will be created. In the other hand, when showing a minus sign (-), Terraform means that a resource will be deleted. Working With Variables Let’s discover how to use Terraform variables to write a cleaner configuration file. We can consider that the AWS region could be variable, that’s why we are going to add this code to the main.tf file: variable "region" { default = "eu-west-3"} variable "region" { default = "eu-west-3"} You can now call it from a Terraform file using: ${var.region} ${var.region} This how our main.tf will look like: variable "region" { default = "eu-west-3"} provider "aws" { region = "${var.region}" shared_credentials_file = "/home/eon01/.aws/credentials" profile = "terraform"} resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } } variable "region" { default = "eu-west-3"} provider "aws" { region = "${var.region}" shared_credentials_file = "/home/eon01/.aws/credentials" profile = "terraform"} resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } } Right ! Let’s do the same thing to “shared_credentials_file” and “profile”: variable "region" { default = "eu-west-3"} variable "shared_credentials_file" { default = "/home/eon01/.aws/credentials"} variable "profile" { default = "terraform"} provider "aws" { region = "${var.region}" shared_credentials_file = "${var.shared_credentials_file}" profile = "${var.profile}"} resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } } variable "region" { default = "eu-west-3"} variable "shared_credentials_file" { default = "/home/eon01/.aws/credentials"} variable "profile" { default = "terraform"} provider "aws" { region = "${var.region}" shared_credentials_file = "${var.shared_credentials_file}" profile = "${var.profile}"} resource "aws_instance" "web" { ami = "ami-0e55e373" instance_type = "t1.micro" tags { Name = "eralabs" } } Using Terraform Maps In my example, I am using the Paris region (eu-west-3) but what if I need to add new regions like Dublin (eu-west1) for instance !? The above code will deploy an EC2 instance to a single region. In order to seolve this problem, the first step to follow here, is finding the AMI we want to use (depending on the region) and then create a variable with the type “map”: variable "my_ami" { type = "map" default = { eu-west-1 = "ami-f90a4880" eu-west-3 = "ami-0e55e373" } description = "I added only 2 regions: Paris and Dublin. You can use as many regions as you want."} variable "my_ami" { type = "map" default = { eu-west-1 = "ami-f90a4880" eu-west-3 = "ami-0e55e373" } description = "I added only 2 regions: Paris and Dublin. You can use as many regions as you want."} According to the used region Terraform should create an EC2 machine with a different AMI. This is done by changing the old AMI line by changing ami = "ami-0e55e373" to ami = "${lookup(var.my_ami, var.region)}" . ami = "ami-0e55e373" ami = "${lookup(var.my_ami, var.region)}" To test this, type terraform plan and you will get this output: terraform plan + aws_instance.web id: ami: "ami-0e55e373" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: + aws_instance.web id: ami: "ami-0e55e373" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: If you manually change the region to eu-west-1 , you will notice that terraform plan will use the other AMI: eu-west-1 terraform plan + aws_instance.web id: ami: "ami-f90a4880" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: + aws_instance.web id: ami: "ami-f90a4880" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: Using Input Arguments In the latest example we changed the value of the region from “eu-west-3” to “eu-west-1” manually. The goal was testing if the map function was working right. In practice, you don’t need to manually change your main.tf file, but you can override the value of region by using a new region as an argument: main.tf Try using “eu-west-1” instead of the default value “eu-west-3”: terraform plan -var region=eu-west-1 terraform plan -var region=eu-west-1 It is possible to input other variables in the same line. As an example, we can change the used profile from “terraform” to “default” using the following command: terraform plan -var region=eu-west-1 -var profile=default terraform plan -var region=eu-west-1 -var profile=default Using Variable Files We want to separate the configuration from the execution code, that’s why we are going to create a file containing the variables we are using (variables.tfvars): region = "eu-west-1" shared_credentials_file = "/home/eon01/.aws/credentials"profile = "terraform" my_ami = { "eu-west-1" = "ami-f90a4880" "eu-west-3" = "ami-0e55e373" } region = "eu-west-1" shared_credentials_file = "/home/eon01/.aws/credentials"profile = "terraform" my_ami = { "eu-west-1" = "ami-f90a4880" "eu-west-3" = "ami-0e55e373" } After removing the variables from the main.tf file, this is how it becomes: main.tf variable "region" {}variable "shared_credentials_file" {}variable "profile" {}variable "my_ami" { type = "map"}provider "aws" { region = "${var.region}" shared_credentials_file = "${var.shared_credentials_file}" profile = "${var.profile}"} resource "aws_instance" "web" { ami = "${lookup(var.my_ami, var.region)}" instance_type = "t1.micro" tags { Name = "eralabs" } } variable "region" {}variable "shared_credentials_file" {}variable "profile" {}variable "my_ami" { type = "map"}provider "aws" { region = "${var.region}" shared_credentials_file = "${var.shared_credentials_file}" profile = "${var.profile}"} resource "aws_instance" "web" { ami = "${lookup(var.my_ami, var.region)}" instance_type = "t1.micro" tags { Name = "eralabs" } } Let’s execute the plan command to see if there the EC2 machine will be created: terraform plan -var-file=variables.tfvars terraform plan -var-file=variables.tfvars Terraform Apply In order to execute create our EC2 machine, we need to execute: terraform apply . Because we are using a file to store our variables, we need to execute: terraform apply terraform apply -var-file=variables.tfvars terraform apply -var-file=variables.tfvars After executing the command above, we will have a similar output to this one: An execution plan has been generated and is shown below.Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + aws_instance.web id: ami: "ami-f90a4880" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. An execution plan has been generated and is shown below.Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + aws_instance.web id: ami: "ami-f90a4880" associate_public_ip_address: availability_zone: ebs_block_device.#: ephemeral_block_device.#: get_password_data: "false" instance_state: instance_type: "t1.micro" ipv6_address_count: ipv6_addresses.#: key_name: network_interface.#: network_interface_id: password_data: placement_group: primary_network_interface_id: private_dns: private_ip: public_dns: public_ip: root_block_device.#: security_groups.#: source_dest_check: "true" subnet_id: tags.%: "1" tags.Name: "eralabs" tenancy: volume_tags.%: vpc_security_group_ids.#: Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. You will be asked to confirm, enter “yes”: Enter a value: yes aws_instance.web: Creating... ami: "" => "ami-f90a4880" associate_public_ip_address: "" => " " availability_zone: "" => " " ebs_block_device.#: "" => " " ephemeral_block_device.#: "" => " " get_password_data: "" => "false" instance_state: "" => " " instance_type: "" => "t1.micro" ipv6_address_count: "" => " " ipv6_addresses.#: "" => " " key_name: "" => " " network_interface.#: "" => " " network_interface_id: "" => " " password_data: "" => " " placement_group: "" => " " primary_network_interface_id: "" => " " private_dns: "" => " " private_ip: "" => " " public_dns: "" => " " public_ip: "" => " " root_block_device.#: "" => " " security_groups.#: "" => " " source_dest_check: "" => "true" subnet_id: "" => " " tags.%: "" => "1" tags.Name: "" => "eralabs" tenancy: "" => " " volume_tags.%: "" => " " vpc_security_group_ids.#: "" => " "aws_instance.web: Still creating... (10s elapsed)aws_instance.web: Creation complete after 19s (ID: i-055aaa2cab2436ab4) Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Enter a value: yes aws_instance.web: Creating... ami: "" => "ami-f90a4880" associate_public_ip_address: "" => " " availability_zone: "" => " " ebs_block_device.#: "" => " " ephemeral_block_device.#: "" => " " get_password_data: "" => "false" instance_state: "" => " " instance_type: "" => "t1.micro" ipv6_address_count: "" => " " ipv6_addresses.#: "" => " " key_name: "" => " " network_interface.#: "" => " " network_interface_id: "" => " " password_data: "" => " " placement_group: "" => " " primary_network_interface_id: "" => " " private_dns: "" => " " private_ip: "" => " " public_dns: "" => " " public_ip: "" => " " root_block_device.#: "" => " " security_groups.#: "" => " " source_dest_check: "" => "true" subnet_id: "" => " " tags.%: "" => "1" tags.Name: "" => "eralabs" tenancy: "" => " " volume_tags.%: "" => " " vpc_security_group_ids.#: "" => " "aws_instance.web: Still creating... (10s elapsed)aws_instance.web: Creation complete after 19s (ID: i-055aaa2cab2436ab4) Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Terraform & Immutable Infrastructure To simply define this concept, an immutable resource or component is replaced for every deployment. For instance, servers are never modified after the deployment. When an updated is needed, a new server should be created from a base/common image with the new updates. In order to see this in practice, I made it explicit to forget adding the SSH key to the EC2 description file, without it you can create an EC2 machine but you can’t access it using SSH. Let’s now add a key pair to the EC2 machine: In the main.tf file: resource "aws_instance" "web" { ami = "${lookup(var.my_ami, var.region)}" instance_type = "t1.micro" key_name = "${var.key_name}" tags { Name = "eralabs" } resource "aws_instance" "web" { ami = "${lookup(var.my_ami, var.region)}" instance_type = "t1.micro" key_name = "${var.key_name}" tags { Name = "eralabs" } In variables.tfvs file: key_name = "my_key.kp" key_name = "my_key.kp" Now you can execute the plan command than the apply command and you will notice that Terraform will not update the machine to add the new key but will destroy it and create a new one with a new configuration: -/+ aws_instance.web (new resource required) id: "i-055aaa2cab2436ab4" => (forces new resource) ami: "ami-f90a4880" => "ami-f90a4880" associate_public_ip_address: "true" => availability_zone: "eu-west-1a" => ebs_block_device.#: "0" => ephemeral_block_device.#: "0" => get_password_data: "false" => "false" instance_state: "running" => instance_type: "t1.micro" => "t1.micro" ipv6_address_count: "" => ipv6_addresses.#: "0" => key_name: "" => "my_key.kp" (forces new resource) network_interface.#: "0" => network_interface_id: "eni-ddcf2fd9" => password_data: "" => placement_group: "" => primary_network_interface_id: "eni-ddcf2fd9" => private_dns: "ip-172-31-43-75.eu-west-1.compute.internal" => private_ip: "172.31.43.75" => public_dns: "ec2-52-211-29-100.eu-west-1.compute.amazonaws.com" => public_ip: "52.211.29.100" => root_block_device.#: "1" => security_groups.#: "1" => source_dest_check: "true" => "true" subnet_id: "subnet-7f510f27" => tags.%: "1" => "1" tags.Name: "eralabs" => "eralabs" tenancy: "default" => volume_tags.%: "0" => vpc_security_group_ids.#: "1" => Plan: 1 to add, 0 to change, 1 to destroy. -/+ aws_instance.web (new resource required) id: "i-055aaa2cab2436ab4" => (forces new resource) ami: "ami-f90a4880" => "ami-f90a4880" associate_public_ip_address: "true" => availability_zone: "eu-west-1a" => ebs_block_device.#: "0" => ephemeral_block_device.#: "0" => get_password_data: "false" => "false" instance_state: "running" => instance_type: "t1.micro" => "t1.micro" ipv6_address_count: "" => ipv6_addresses.#: "0" => key_name: "" => "my_key.kp" (forces new resource) network_interface.#: "0" => network_interface_id: "eni-ddcf2fd9" => password_data: "" => placement_group: "" => primary_network_interface_id: "eni-ddcf2fd9" => private_dns: "ip-172-31-43-75.eu-west-1.compute.internal" => private_ip: "172.31.43.75" => public_dns: "ec2-52-211-29-100.eu-west-1.compute.amazonaws.com" => public_ip: "52.211.29.100" => root_block_device.#: "1" => security_groups.#: "1" => source_dest_check: "true" => "true" subnet_id: "subnet-7f510f27" => tags.%: "1" => "1" tags.Name: "eralabs" => "eralabs" tenancy: "default" => volume_tags.%: "0" => vpc_security_group_ids.#: "1" => Plan: 1 to add, 0 to change, 1 to destroy. Using Terraform Modules Terraform hots a public registry where you can find common reusable modules. a public registry You can use this registry modules for your projects. Some of them are verified by HashiCorp, the company behind Terraform. Some of them In order to put this in practice, we are going to do the same operation(creating an EC2 machine), but using this module . this module Create a new file (for example: main2.tf) and add these lines: module "ec2_cluster" { source = "terraform-aws-modules/ec2-instance/aws" name = "my-cluster" instance_count = 5 ami = "ami-ebd02392" instance_type = "t2.micro" key_name = "user1" monitoring = true vpc_security_group_ids = ["sg-12345678"] subnet_id = "subnet-eddcdzz4" tags = { Terraform = "true" Environment = "dev" }} module "ec2_cluster" { source = "terraform-aws-modules/ec2-instance/aws" name = "my-cluster" instance_count = 5 ami = "ami-ebd02392" instance_type = "t2.micro" key_name = "user1" monitoring = true vpc_security_group_ids = ["sg-12345678"] subnet_id = "subnet-eddcdzz4" tags = { Terraform = "true" Environment = "dev" }} Now type terraform init and the module files will be downloaded. You can use the plan then the apply command. terraform init Connect Deeper In this tutorial, we started manipulating Terraform with AWS but this is an introduction and it will be extended in Practical AWS online training . If you are interested in Practical AWS training, you can make an order and start learning AWS right now. If you are interested in Practical AWS training, you can make an order and start learning AWS right now. Practical AWS training make an order You can also download my mini ebook 8 Great Tips to Learn AWS. You can also download my mini ebook 8 Great Tips to Learn AWS. 8 Great Tips to Learn AWS