Serverless Stack — CI/CD — Blue-Green Deployments

Check out this awesome Serverless Stack with built-in CI/CD and Blue-Green Deployments

I love building CloudFormation stacks, crazy I know… I also love serverless event-driven architectures, who doesn’t…

I wanted to create a reusable stack that I could easily use to build web applications.

The stack consists of an API, UI, and Async Tasks. This isn’t the cool part.

The cool part is the built-in CI/CD via CodePipeline and Blue-Green deployments of Lambda.

CI/CD and Blue-Green deployments are very important for the stability and health of an application. It is important to deliver often and fail fast.

Code

The code. Check it out, run it, and let me know what you think…

thestackshack/serverless-stack-cicd_serverless-stack-cicd - Serverless stack with CI/CD & blue-green deployments - API + Static UI + Async Tasks_github.com

Architecture

Lets take a look at the architecture before we talk about the CI/CD and Blue-Green deployments.

CI/CD

When you push code or infrastructure changes to this application CodePipeline builds and updates everything. Your infrastructure stacks are updated. Your code is built, tested, and deployed.

You can make changes and push all day long. Development will be fast and predictable. Happy days.

Oh, and there is a sandbox and a prod environment, so you can test out your changes in sandbox prior to pushing them to prod. It’s nice to be able to test changes in a sandbox environment with your team before you go live with them.

Here is what the pipeline looks like:

Lets take a closer look at each pipeline stage and action. A pipeline consists of stages. Each stage consists of actions.

How you organize your stages and actions depends on 2 things. The order you want your stages and actions to be processed in, and the stage input and output dependencies.

Stages can be organized in series or in parallel or both. This is defined by the stage input and output artifacts.

Actions can also be organized in series or in parallel or both. This is defined by the action order property and the input and output artifacts.

If the input artifact for each stage is the output artifact for the previous stage then they will be in series. If the input artifact for a stage is the output artifact of several previous stages, and those previous stages don’t depend on each other, then those previous stages will be in parallel.

Enough about CodePipeline, lets learn more about our pipeline:

Source Stage — GitHub Push

The source stage is triggered on every GitHub push.

- Name: SourceActions: - Name: CloneRepositoryActionTypeId:Category: SourceOwner: ThirdPartyVersion: 1Provider: GitHubOutputArtifacts: - Name: GitSourceConfiguration:Owner: !Ref GitHubOwnerBranch: 'master'Repo: !Ref GitHubRepoOAuthToken: !Ref GitHubTokenRunOrder: 1

The prod pipeline uses the master GitHub branch and the sandbox pipeline uses the develop GitHub branch.

Tasks Package Stage

This stage uses CodeBuild to build & test the Lambda, then package the CloudFormation template.

- Name: TasksPackageActions: - Name: PackageActionTypeId:Category: BuildOwner: AWSProvider: CodeBuildVersion: 1Configuration:ProjectName: !Ref TasksCodeBuildProdInputArtifacts: - Name: GitSourceOutputArtifacts: - Name: TasksOutputRunOrder: 1

Our Lambda within the CloudFormation template uses a relative CodeUri property. When using the CodeUri property you have to use CloudFormation package(aws cloudformation package). CloudFormation package does 2 things. First it uploads your Lambda at the CodeUri location as a zip file to S3, then it exports the template with the CodeUri replaced with the zip file location in S3.

Here is the entire CodeBuild buildspec.yml.

version: 0.2phases:install:commands:pre_build:commands: - echo Installing source NPM dependencies...- cd ./stacks/tasks && npm installbuild:commands: - echo Testing the code- npm test- echo Removing dev dependencies- rm -Rf node_modules- npm install --productionpost_build:commands: - aws cloudformation package --template-file tasks.stack.yml --s3-bucket ${Bucket} --output-template-file tasks.stack.output.ymlartifacts:base-directory: 'stacks/tasks'type: zipfiles: - tasks.stack.output.yml

This CodeBuild exports the packaged CloudFormation template. This will be used in the next stage.

Tasks Stack Stage

This stage mutates (create/update) the tasks stack.

- Name: TasksStackActions: - Name: CreateChangeSetInputArtifacts: - Name: TasksOutputActionTypeId:Category: DeployOwner: AWSVersion: 1Provider: CloudFormationConfiguration:TemplatePath: "TasksOutput::tasks.stack.output.yml"ActionMode: CHANGE_SET_REPLACECapabilities: CAPABILITY_NAMED_IAMRoleArn: !GetAtt CloudFormationRole.ArnStackName: !Sub "${AWS::StackName}-tasks-prod"ChangeSetName: !Sub "${AWS::StackName}-tasks-prod-cs"RunOrder: 1- Name: ExecuteChangeSetInputArtifacts: - Name: TasksOutputActionTypeId:Category: DeployOwner: AWSVersion: 1Provider: CloudFormationConfiguration:ActionMode: CHANGE_SET_EXECUTECapabilities: CAPABILITY_NAMED_IAMRoleArn: !GetAtt CloudFormationRole.ArnStackName: !Sub "${AWS::StackName}-tasks-prod"ChangeSetName: !Sub "${AWS::StackName}-tasks-prod-cs"RunOrder: 2

The stage has two actions.

The first action creates a CloudFormation ChangeSet. You can see that the input artifact is the output artifact of the Tasks Package stage. Take a look at the TemplatePath, it is from the CodeBuild package command.

The second action executes the CloudFormation ChangeSet mutating the stack.

API Package & API Stack Stages

These stages are exactly the same as the Tasks stages. They package the Lambda & CloudFormation template, then create the CloudFormation ChangeSet, and finally mutate that ChangeSet.

UI Stack Stage

The UI stack stage does things in the reverse order. Instead of building, testing, and uploading the code, then mutating the stack. It mutates the stack and then builds, tests, and uploads the code.

- Name: UIActions: - Name: StackInputArtifacts: - Name: GitSourceActionTypeId:Category: DeployOwner: AWSVersion: 1Provider: CloudFormationConfiguration:TemplatePath: "GitSource::stacks/ui/ui.stack.yml"ActionMode: CREATE_UPDATECapabilities: CAPABILITY_NAMED_IAMRoleArn: !GetAtt CloudFormationRole.ArnStackName: !Sub "${AWS::StackName}-ui-prod"ParameterOverrides: !Sub |{"Domain": "${Domain}","TLD" : "${TLD}"}RunOrder: 1- Name: DeployActionTypeId:Category: BuildOwner: AWSProvider: CodeBuildVersion: 1Configuration:ProjectName: !Ref UICodeBuildProdInputArtifacts: - Name: GitSourceRunOrder: 2

The first action mutates the CloudFormation stack and creates the S3 bucket we need in the next action.

The second action executes a CodeBuild that uploads the code to the S3 bucket thereby deploying the static website.

Here is the CodeBuild buildspec.yml.

version: 0.1phases:install:commands:pre_build:commands:build:commands: - aws s3 sync stacks/ui/www "s3://$(aws cloudformation describe-stacks --stack-name ${StackName} --query "Stacks[0].Outputs[0].OutputValue" --output text)" --acl bucket-owner-full-control --acl public-read --delete --cache-control "max-age=1" --exclude stacks/ui/www/assets- aws s3 sync stacks/ui/www/assets "s3://$(aws cloudformation describe-stacks --stack-name ${StackName} --query "Stacks[0].Outputs[0].OutputValue" --output text)/assets" --acl bucket-owner-full-control --acl public-read --delete --cache-control "max-age=31536000"post_build:commands:

This is interesting. We get the S3 bucket name from the previous step by fetching the output parameter from the stack. Here is the command:

$(aws cloudformation describe-stacks--stack-name ${StackName}--query "Stacks[0].Outputs[0].OutputValue"--output text)

That’s it. That’s how the pipeline performs CI/CD for our infrastructure and code.

Now lets take a look at the blue-green deployments.

Blue-Green Deployments

Within our api CloudFormation template we have a Lambda.

LambdaFunction:Type: AWS::Serverless::FunctionProperties:Handler: index.handlerTimeout: 5Role: !GetAtt IamRoleLambdaExecution.ArnCodeUri: ./Runtime: nodejs6.10AutoPublishAlias: liveDeploymentPreference:Type: Canary10Percent5MinutesAlarms: - !Ref 5xxAlarm- !Ref 4xxAlarm- !Ref LatencyAlarmEnvironment:Variables:TasksSnsTopic:Fn::ImportValue: !Sub "${TasksStack}-SNSTopic"

This Lambda uses the Serverless Application Model (SAM), which is a CloudFormation transformer.

When using SAM types within a CloudFormation template you need to add the transform definition. **Transform**: AWS::Serverless-2016–10–31.

I’m not sure why SAM exists at all. Why wasn’t CloudFormation extended to include these new SAM features? I read somewhere that SAM is less verbose. Is that the only reason? If so that doesn’t make up for the fragmentation and confusion SAM brings. Just my 2 cents…

Take a look at the DeploymentPreference property. This is where we define the Safe Traffic Shifting.

We use Canary10Percent5Minutes which routes 10% of traffic to the new Lambda, then waits 5 minutes, if all is good (no alarms go off) the remaining traffic is routed and the deployment is done. There are 3 types of traffic shifting.

• LinearXPercentYMinutes: Traffic to new version will linearly increase in steps of X percentage every Y minutes. Ex: Linear10PercentEvery10Minutes will add 10 percentage of traffic every 10 minute to complete in 100 minutes.
• CanaryXPercentYMinutes: X percent of traffic will be routed to new Version once, and wait for Y minutes in this state before sending 100 percent of traffic to new version. Some people call this as Blue/Green deployment. Ex: Canary10Percent15Minutes will send 10 percent traffic to new version and 15 minutes later complete deployment by sending all traffic to new version.
• AllAtOnce: This is an instant shifting of 100% of traffic to new version. This is useful if you want to run run pre/post hooks but don’t want a gradual deployment. If you have a pipeline, you can set Beta/Gamma stages to deploy instantly because the speed of deployments matter more than safety here.

The alarms property is where you define CloudWatch alarms to monitor while your traffic is being shifted. If any of these alarms go off the deployment will be reverted.

Alarms: - !Ref 5xxAlarm

• !Ref 4xxAlarm
• !Ref LatencyAlarm

Our alarms are based on the API Gateway traffic. Are we getting too many 5xx or 4xx errors, or did the latency spike? If so something is wrong with our new version and we don’t want to deploy it.

Here are the alarms.

5xxAlarm:Type: AWS::CloudWatch::AlarmDependsOn: RestApiProperties:AlarmDescription: 5xx alarm for api gatewayNamespace: 'AWS/ApiGateway'MetricName: 5XXErrorDimensions: - Name: ApiNameValue: !Ref RestApiStatistic: SumPeriod: '60'EvaluationPeriods: '3'Threshold: '10'ComparisonOperator: GreaterThanOrEqualToThreshold4xxAlarm:Type: AWS::CloudWatch::AlarmDependsOn: RestApiProperties:AlarmDescription: 4xx alarm for api gatewayNamespace: 'AWS/ApiGateway'MetricName: 4XXErrorDimensions: - Name: ApiNameValue: !Ref RestApiStatistic: SumPeriod: '60'EvaluationPeriods: '3'Threshold: '10'ComparisonOperator: GreaterThanOrEqualToThresholdLatencyAlarm:Type: AWS::CloudWatch::AlarmDependsOn: RestApiProperties:AlarmDescription: latency alarm for api gatewayNamespace: 'AWS/ApiGateway'MetricName: LatencyDimensions: - Name: ApiNameValue: !Ref RestApiStatistic: AveragePeriod: '60'EvaluationPeriods: '3'Threshold: '25000'ComparisonOperator: GreaterThanOrEqualToThreshold

That’s it. I hope you enjoyed this post and get some use out of this stack.

Please clap if you liked this article.