330 reads

GIT-version Your AWS CloudFormation Parameters using Stackuchin

by Marius MitrofanJune 29th, 2020

Too Long; Didn't Read

Stackuchin is a CLI tool developed by Rungutan and designed to automatically create, update and delete CloudFormation stacks in multiple AWS accounts and regions at the same time. The tool is available for SMBs and is 100% Serverless, API driven, available to SMBs. It can be run either manually or through a pipeline definition in your CI/CD system. We use Stackuchin to handle updates for around 3000 AWS CloudFormations. We have around 200 stacks on average deployed in each and every of the 15 regions our platform currently supports.

Companies Mentioned

Coin Mentioned

featured image - GIT-version Your AWS CloudFormation Parameters using Stackuchin

Full disclosure: I'm the CTO of https://rungutan.com - the first API Load Testing SaaS platform, 100% Serverless, API driven, finally available for SMBs, the company which has built this open-source software - Stackuchin.

What is Stackuchin?

It's a CLI tool developed by Rungutan and designed to automatically create, update and delete CloudFormation stacks in multiple AWS accounts and regions at the same time.

Why use the CLI?

This CLI has been designed for:

versioning AWS CloudFormation parameters in GIT
deploying to multiple AWS accounts or AWS regions either in PARALLEL or SEQUENTIAL
send notifications to Slack channels with AWS CloudFormation errors
support create, update and delete commands
can be ran either manually or through a pipeline definition in your CI/CD system
supports parent/child stacks
supports NoEcho parameters
supports tagging of resources at stack level
supports unattended deployment (through a CI/CD system)
supports both JSON and YAML versions of AWS CloudFormation

And this is just the tip of the iceberg...

Is it production ready?

We, at Rungutan, in order to support global concurrency for load testing and ensure high availability as well, have around 200 stacks on average deployed in each and every of the 15 regions our platform currently supports.

In short, yes, we use Stackuchin to handle updates for around 3000 AWS CloudFormation stacks.

And no, we're not exagerating or bumping the numbers :-)

What are the normal use cases?

If simply the fact that you can now git-version all your stacks AND their stack parameters, isn't enough, then:

your developers can now manage AWS CloudFormation stack themselves, WITHOUT needing to have any "write" IAM permissions
you can use CI/CD for automated deployments
you can use pull requests to review parameter/stack changes

How to install the CLI?

pip install stackuchin

How to run the CLI?

Check the overall help menu

$ stackuchin help

usage: stackuchin <command> [<args>]

To see help text, you can run:
    stackuchin help
    stackuchin version
    stackuchin create --help
    stackuchin delete --help
    stackuchin update --help
    stackuchin pipeline --help

CLI tool to automatically create, update and delete AWS CloudFormation stacks in multiple AWS accounts and regions at the same time

positional arguments:
  command     Command to run

optional arguments:
  -h, --help  show this help message and exit

Check the help menu for a specific command

$ stackuchin create --help

usage: stackuchin <span class="o" style="box-sizing: inherit;">[</span>-h<span class="o" style="box-sizing: inherit;">]</span> <span class="o" style="box-sizing: inherit;">[</span>--stack_file STACK_FILE<span class="o" style="box-sizing: inherit;">]</span> --stack_name STACK_NAME <span class="o" style="box-sizing: inherit;">[</span>--secret <span class="nv" style="box-sizing: inherit; color: rgb(0, 120, 189);">Parameter</span><span class="o" style="box-sizing: inherit;">=</span>Value<span class="o" style="box-sizing: inherit;">]</span> <span class="o" style="box-sizing: inherit;">[</span>--slack_webhook SLACK_WEBHOOK<span class="o" style="box-sizing: inherit;">]</span> <span class="o" style="box-sizing: inherit;">[</span>--s3_bucket S3_BUCKET<span class="o" style="box-sizing: inherit;">]</span> <span class="o" style="box-sizing: inherit;">[</span>--s3_prefix S3_PREFIX<span class="o" style="box-sizing: inherit;">]</span> <span class="o" style="box-sizing: inherit;">[</span>-p PROFILE<span class="o" style="box-sizing: inherit;">]</span>

Create <span class="nb" style="box-sizing: inherit; color: rgb(0, 45, 71);">command</span> system

optional arguments:
  -h, --help            show this <span class="nb" style="box-sizing: inherit; color: rgb(0, 45, 71);">help</span> message and <span class="nb" style="box-sizing: inherit; color: rgb(0, 45, 71);">exit</span>
  --stack_file STACK_FILE
                        The YAML file which contains your stack definitions.
                        Defaults to <span class="s2" style="box-sizing: inherit; color: rgb(213, 45, 64);">"./cloudformation-stacks.yaml"</span> <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">if</span> not specified.
  --stack_name STACK_NAME
                        The stack that you wish to create
  --secret <span class="nv" style="box-sizing: inherit; color: rgb(0, 120, 189);">Parameter</span><span class="o" style="box-sizing: inherit;">=</span>Value
                        Argument used to specify values <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">for</span> NoEcho parameters in your stack
  --slack_webhook SLACK_WEBHOOK
                        Argument used to overwrite environment variable STACKUCHIN_SLACK.
                        If argument is specified, any notifications will be sent to this URL.
                        If not specified, the script will check <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">for</span> env var STACKUCHIN_SLACK.
                        If neither argument nor environment variable is specified, <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">then</span> no notifications will be sent.
  --s3_bucket S3_BUCKET
                        Argument used to overwrite environment variable STACKUCHIN_BUCKET_NAME.
                        If argument is specified, <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">then</span> the template is first uploaded here before used in the stack.
                        If not specified, the script will check <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">for</span> env var STACKUCHIN_BUCKET_NAME.
                        If neither argument nor environment variable is specified, <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">then</span> the script will attempt to feed the template directly to the AWS API call, however, due to AWS CloudFormation API call limitations, you might end up with a bigger template in byte size than the max value allowed by AWS.
                        Details here -&gt; https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html
  --s3_prefix S3_PREFIX
                        Argument used to overwrite environment variable STACKUCHIN_BUCKET_PREFIX.
                        The bucket prefix path to be used when the S3 bucket is defined.
  -p PROFILE, --profile PROFILE
                        The AWS profile you<span class="err" style="box-sizing: inherit; color: rgb(130, 26, 38);">'</span>ll be using.
                        If not specified, the <span class="s2" style="box-sizing: inherit; color: rgb(213, 45, 64);">"default"</span> profile will be used. 
                        If no profiles are defined, <span class="k" style="box-sizing: inherit; color: rgb(0, 45, 71);">then</span> the default AWS credential mechanism starts.

How do you actually use it ?

The logic of the app is simple:

Specify the operation that you want to perform
Specify the file which contains the parameters for your stack
Add any secrets (aka NoEcho parameters) that your stack might need (optional)

Here's the most basic simple definition of a "stack file":

your-first-stack:
  Account: 123112321123
  Region: us-east-1
  Template: cloudformation-template.yaml
  # All parameters except NoEcho.
  Parameters:
    paramA: valA
  Tags:
    Environment: UTILITIES
    Team: DevOps
    MaintainerEmail: [email protected]
    MaintainerTeam: Rungutan

another-stack-name:
  Account: 123112321123
  Region: us-east-1
  Template: some-folder/cloudformation-some-other-template.yaml
  # Stack without readable parameters.
  Parameters: {}
  Tags:
    Environment: UTILITIES
    Team: DevOps
    MaintainerEmail: [email protected]
    MaintainerTeam: Rungutan

Running it as a pipeline

cat > input.yaml <<EOL

pipeline:
  update:
    - stack_name: TestUpdateStack
  delete:
    - stack_name: TestDeleteStack
  create:
    - stack_name: TestCreateStack
      secrets:
        - Name: SomeSecretName
          Value: SomeSecreValue
EOL

stackuchin  pipeline --pipeline_file input.yaml

Get alerts in Slack

Use the environment variable STACKUCHIN_SLACK or the argument --slack_webhook to specify a Slack incoming webhook to push your alerts.

You get notified ALL with PROPER MESSAGES, so that you wouldn't need to have to open your AWS Console to fix your stuff.

Here's a sample:

Running it in a CI/CD process

Here's a sample pipeline that uses our official Docker image to run it in using GitLab CI/CD:

image: rungutancommunity/stackuchin:latest

stages:
  - deploy_updates

variables:
  AWS_DEFAULT_REGION: us-east-1
  STACKUCHIN_SLACK: https://hooks.slack.com/services/some_slack_webhook
  STACKUCHIN_BUCKET_NAME: some-deployment-bucket-in-us-east-1
  STACKUCHIN_BUCKET_PREFIX: some/prefix/this/is/optional

deploy_updates:
  only:
    refs:
      - master
  stage: deploy_updates
  script:
    - |
        cat > pipeline.yaml <<EOF
        pipeline:
          pipeline_type: parallel
          update:
            - stack_name: My-First-Stack
            - stack_name: My-Second-Stack
        EOF
    - stackuchin pipeline --stack_file stack_file.yaml --pipeline_file pipeline.yaml

Notes

This app currently does *NOT* support contracted forms of verbs in AWS CloudFormation when using YAML templates.

In short, if your AWS CF templates written in YAML use stuff like !If, then you have to update them to use their respective version -> Fn::If.