How We Manage Our Infrastructure: Intro to Stacker

Stacker is Remind’s solution to deploying cloud infrastructure at scale. It allows you to quickly build, update, validate and deploy AWS cloudformation templates. It is built on top of the popular troposphere library which gives you the full power of python when writing your templates. Stacker is a production ready solution which Remind uses to deploy our and is used at many other companies at an even larger scale. 100+ stacks This is the first part in a series of blog posts demoing Stacker. Introduction to Cloudformation When you are managing cloud infrastructure, version control and keeping track of changes can be very challenging. This task becomes even harder when you need to replicate your entire production stack multiple times for development and testing purposes. You can try writing your own scripts to create all the resources, but those quickly get complex when you have many resources depending on each other and you need to figure out the correct creation order. To address this, Amazon created Cloudformation, a way for you to write customizable low-level templates that you can send to Amazon to automatically create all the resources for you. Stack: A stack is a collection of AWS resources that you can manage as a single unit. In other words, you can create, update, or delete a collection of resources by creating, updating, or deleting stacks. All the resources in a stack are defined by the stack’s AWS CloudFormation template. The Tao of Stacker Let’s start with a story of how we started building Stacker and what design decisions were made along the way. This should give you some insight into the proper way to leverage Stacker and get the most out of it. Chapter 1: The Monolith We started with a monolith Stack, one single template which created all of our AWS resource. Life was simple, everything was accessible from one file and there was no need for Stacks to communicate with each other. However, this approach had several problems: We were updating all of our infrastructure at once, this took a long time and if the Stack execution ever failed it would go into Rollback mode. This means that if one resources failed to create it would have to revert the changes on all the other resources. Even worse, if it got into a bad rollback state it would get stuck and your production Stack would need to be deleted and recreated. That is not a fun process! It was dangerous: At one point our template was over 8000+ lines of code, it was very finicky to deal with especially when every change affects your mission critical infrastructure. It was complicated: Chapter 2: The Nested Monolith We were super excited when Nested Cloudformation Stacks came out. We thought that it would solve all the previous problems we had, unfortunately it didn’t: The Rollback problem would still occur with nested Stacks because everything was still submitted as a single template. It was still dangerous: The Cloudformation template would still end up long and complicated, making changes difficult to reason about. It was still complicated: Chapter 3: Present — Modular Stacks The current version of Stacker works by allowing you to generate many separate Stacks. Each Stack represents an independent piece of infrastructure. Cross Stack references which was the biggest hurdle in implementing this approach are created by using the “Output Lookup” feature which we introduced early on. Here are some guiding principles that we believe in: Each separate piece of infrastructure can be represented as its own Stack. This solves the Rollback problem because when a certain piece of infrastructure fails, it only affects that isolated resource and not the whole production stack, this make it much easier and safer to resolve issues. In addition, this allows us to create premade templates for commonly deployed Stacks. You can view dozens of custom templates we created to make your life easier at the repo. Small Composable Stacks: stacker_blueprints When deploying a Stack you specify a configuration file and an environment file. This allows you to modify the Stacks based on which environment you are launching into. For example, we have separate environment files for and which use different passwords and staging uses less expensive resources to save money. Swappable Environments: staging production Lot’s of other infrastructure management tools support multiple vendors. We made the very explicit decision to support exclusively AWS. AWS makes great reliable software and supporting only them allows us to make the application code simpler, it allows us to create better plugins, and allows us to offload a lot of logic to AWS itself. If you are worried about vendor lock-in there are much bigger obstacles to switching providers than Stacker. AWS Exclusive: Stacker Features Powerful templates written in Python Cloudformation templates are written in JSON or YAML. Yuck! These are simple configuration languages that don’t really give you much to work with. Stacker lets you build your templates using the library which gives you the full power of Python. Troposphere You can use and to choose when to create or not create certain resources. You can use to avoid having to replicate the same thing 100 times. You can pass around which help you create different types of resources easily. conditionals variables loops functions Stack Validation Our use of Troposphere allows Stacker to check that your stacks are valid before you submit to the cloud, allowing you to fix things much more quickly and safeguarding you from any accidental changes. In addition, if something fails while the resources are being created in the cloud it outputs the error directly to your console, so you can see everything in one place and don’t have to keep opening your browser. The workflow of creating a template, validating the syntax, submitting it to AWS, waiting for something to fail, rinse & repeat is horrendously time consuming - Ben Whaley, whaletech.co Helpful add-ons In addition we can write custom add-ons for Stacker that help with commonly performed tasks. There are two kinds of “plugins” that one can write for stacker. Hooks These allow you to run things before or after you deploy all your stacks. To illustrate this, let’s take the very common take of creating an EC2 instance that you can SSH into. It’s very annoying to have go to the AWS console, create a keypair and then copy around secret information into your template. The keypair hook automatically does all this for you, plus it is more secure because it generates a new secret every time and this information never leaves the cloud. Other useful hooks include: Prompts you to import or use an existing keypair, before the Stack is run. Keypair: Allows you to create a custom Route53 domain to be used within your template. Route53: Builds Lambda payloads from user configuration and uploads them to S3. Lambda: We made it super easy for you to create and plug in your own hooks. Custom: Lookups: Lookups are slightly tricky to wrap your head around, but a very useful tool when you understand. For example, imagine you create a S3 Bucket and you want to create another Stack that uses this S3 bucket in someway. You can do that using what we call an output lookup which allows you to grab the information about that S3 bucket and inject into another Stack. Again, we have lots of useful lookups: The output lookup is the one that was just mentioned above and allows you to access outputs of other Stacks. Output: The KMS lookup allows you to put encrypted sensitive information such as passwords into your configuration, which will then be decrypted using KMS. KMS: This allows you to pass in a file to your stack, most frequently used for UserData content. File: We made it super easy for you to create and plug in your own lookups. Custom: Pre-built Blueprints Given that the Stacks are small and composable we can create generic Blueprints for everyone to use. We have open-sourced our public list of generic templates at . These include Blueprints for: stacker_blueprints VPC’s (MySQL, Postgres) RDS Instance Firehose Autoscaling Groups Entire Empire Stack Many more! Give it a try! Stacker is a super easy tool for deploying cloud infrastructure at scale. Try it out and let us know how it works for you. If you see anything that you would like added to Stacker, we are very open to PR’s. Let’s make the world of infrastructure automation better :) Tweet your creations at us: @remindeng Github Repo: https://github.com/remind101/stacker Online Docs: http://stacker.readthedocs.io/en/latest/ Slack Support Channel: https://empire-slack.herokuapp.com/ Originally published at engineering.remind.com by @tau .