As you may have seen, we just launched our of env0 last month. As part of the run-up to the launch, our dev team had to go through and make sure everything about our infrastructure was ready for ongoing public use: one element being creating a for both our Application and our . public beta maintenance mode public API Why do you need a maintenance mode for something like env0, which is built to be highly available? Well whether it’s human error (as in the case of ), a DDOS attack (like with ), or just a major upgrade (like ), even the most highly available applications from the most experienced providers sometimes need to be able to be taken offline for a short period of time. And when you have to do that, you want to make sure that your users understand what is happening and still have a good experience. AWS S3 github Zapier Here’s how our team put together our maintenance mode using Terraform, AWS, and Github Pages (including all the code at the end!) Architecture We host all of our infrastructure on AWS, with a clear separation between Application and API: React front end application is hosted on S3 with CloudFront as a CDN. The Backend services are mostly Serverless using AWS Lambda with API Gateway that manages our public API. DNS is managed by Route 53. Since we have a separation between the frontend application and our API, we need to have a maintenance mode for each of them, especially as we also have a public API that is used by our customers for integration in their CI/CD pipelines and other tools. Don’t forget this if you have multiple ways in which people access your services! Requirements As with any project, the first step is to lay out requirements and constraints. In this case, we came up with the following list of what the solution should do: Be implemented with Infrastructure as Code (IaC) as all of our stack is based on IaC Switch back and forth as fast as possible between normal and maintenance mode Switch automatically between normal and maintenance mode Be hosted by separate providers, where possible, from the current infrastructure in order to provide maximum redundancy Have an internal backdoor to allow our team to deploy a fix and test it before switching back Not override the current configuration with new deployments Communicate to an active user that we are in maintenance mode Implementing a Solution After investigating, we came to a few conclusions: We wanted to have a simple place to store the maintenance mode html, so we went with github pages. The switch itself will be done through the DNS, pointing to github pages when we are in maintenance mode. We will create a new DNS record and a Cloudfront distribution that will be our backdoor, and it will be always up and running. We will use our current push mechanism to notify existing users that are in the app that we are currently in maintenance mode. Based on that, we implemented the following at each piece of the system — each system contains a description of what we’re doing, along with the link to a gist with the actual Terraform code. Cloudfront Looking at the Cloudfront distribution code, you can see that we are creating 2 distributions, one is for the actual application and the other one is for a backdoor: resource { = } resource { = = = = = origin { = = aws_s3_bucket.website_bucket.bucket_domain_name s3_origin_config { = aws_cloudfront_origin_access_identity.origin_access_identity.cloudfront_access_identity_path } } default_cache_behavior { = [ , ] = [ , ] forwarded_values { = = [] cookies { = } } = = // day = // year = = = } restrictions { geo_restriction { = } } viewer_certificate { = count. == ? var.acm_certificate_arn : var.acm_certificate_arn_backdoor = = } = list(count. == ? var.dns_name : var.backdoor_dns_name) } "aws_cloudfront_origin_access_identity" "origin_access_identity" comment " access identity" ${aws_s3_bucket.website_bucket.bucket} "aws_cloudfront_distribution" "website_cdn" count 2 # one for the actual cloudfront and one is for the backdoor enabled true price_class "PriceClass_200" http_version "http2" default_root_object "index.html" origin_id "origin-bucket- " ${aws_s3_bucket.website_bucket.id} domain_name origin_access_identity allowed_methods "GET" "HEAD" cached_methods "GET" "HEAD" query_string false headers forward "none" min_ttl 0 default_ttl 86400 1 max_ttl 31536000 1 target_origin_id "origin-bucket- " ${aws_s3_bucket.website_bucket.id} viewer_protocol_policy "redirect-to-https" compress true restriction_type "none" acm_certificate_arn index 0 ssl_support_method "sni-only" minimum_protocol_version "TLSv1.2_2018" aliases index 0 Route53 Also in AWS, we want to ensure that Route53 is pointing in the right direction. In case we are not in maintenance mode it should point to the Cloudfront distribution, and when in maintenance mode it should point our CNAME to the github page. In either case, we should have the backdoor pointing to the Cloudfront distribution. Additionally, we also set the TTL to be 60 seconds so it won’t take too long to move back and forth between maintenance mode and regular operation: resource { zone_id = var.aws_route53_zone_id name = var.dns_name = records = list(var.maintenance_mode_enabled ? : aws_cloudfront_distribution.website_cdn[0].domain_name) ttl = 60 } { zone_id = var.aws_route53_zone_id name = var.backdoor_dns_name = alias { name = aws_cloudfront_distribution.website_cdn[1].domain_name zone_id = aws_cloudfront_distribution.website_cdn[1].hosted_zone_id evaluate_target_health = } } "aws_route53_record" "dns_record" type "CNAME" " .github.io" ${var.github_organization} resource "aws_route53_record" "dns_backdoor_record" type "A" false ⚠️ Pay attention that this Terraform code does not create the Route53 hosted zone, nor the SSL certificates — you need to complete those as appropriate for your own setup ⚠️ Git Repo Next, we need a git repo containing the html files for the new maintenance mode site. So we’ve created a simple Terraform code that will create the repo as well as add all existing files in the folder to the repo, which in our case is the maintenance mode html file: “maintenance_mode_website” resource { = var.github_repo_name = = = = } resource { = github_repository.maintenance-mode-repo.name = var.github_username = = [github_repository.maintenance-mode-repo] } resource { = github_repository.maintenance-mode-repo.name = { for key, value fileset(path.root, ): => value } // There is a limitation here, Terraform file function supports UTF- files, // so any images other non text files can't be included the folder. = trimprefix(each.value, ) = file(each.value) = [github_repository_collaborator.users_repos] } "github_repository" "maintenance-mode-repo" name description "Maintenance mode repo for " ${var.github_repo_name} private true auto_init true default_branch "master" "github_repository_collaborator" "users_repos" repository username permission "admin" depends_on "github_repository_file" "maintenance-mode-files" repository for_each in "maintenance_mode_website/*.*" value 8 or in file "maintenance_mode_website/" content depends_on Github Pages The last part is the trickiest to implement in Terraform, because Github pages configuration is not actually part of the , which means that I can’t really configure it with the Terraform code. However because github offers an API to , it can be done programmatically via the API. In our case, we can use the env0 to trigger those API calls once the deploy is finished: github Terraform provider configure github pages custom flows feature version: 1 deploy: steps: terraformOutput: after: - scripts/set_maintanance_mode.sh ${TF_VAR_github_organization} "$(terraform output github_repo_name)" "$(terraform output website_endpoint)" ${TF_VAR_github_username} ${TF_VAR_github_token} set -e REPO_ORGANIZATION=$1 REPO_NAME=$2 CNAME=$3 GITHUB_USERNAME=$4 GITHUB_TOKEN=$5 if [[ ${REPO_NAME} != "" && CNAME != "" ]]; then echo "Going to set environment to maintenance mode, repo name: ${REPO_NAME}, CNAME: ${CNAME}" GIT_AUTH="${GITHUB_USERNAME}:${GITHUB_TOKEN}" curl --user "${GIT_AUTH}" \ --request POST \ --header "Content-Type: application/json" \ --header "Accept: application/vnd.github.switcheroo-preview+json" \ --url "https://api.github.com/repos/${REPO_ORGANIZATION}/${REPO_NAME}/pages" \ --data "{ \"source\": { \"branch\": \"master\" } }" curl --user "${GIT_AUTH}" \ --request PUT \ --header "Content-Type: application/json" \ --url "https://api.github.com/repos/${REPO_ORGANIZATION}/${REPO_NAME}/pages" \ --data "{ \"cname\": \"${CNAME}\", \"source\": \"master\" }" else echo "No need to set environment to maintenance mode" fi # !/usr/bin/env bash Deployment Now that our system is all configured, all I have to do is change the Terraform variable of the maintenance mode to be true/false and deploy the environment (in our case via the env0 UI). Conclusion The complete template source code can be found in this which includes all the Terraform code, scripts, our env0.yml and the maintenance page html file. We hope you find this useful, or get other ideas for more ways to use Terraform for your deployment workflows. Our next blog post will give you a sneak preview on how we are creating a maintenance mode on our API using Terraform. github repo About env0 env0 lets your team manage their own environments in AWS, Azure and Google, governed by your policies and with complete visibility & cost management. You can learn more about env0 and you can also . Feel free to drop us your thoughts below! here try it out yourself
