One of the benefits of serverless is the pay-per-use pricing model you get from the platform. That is, if your code doesn’t run, you don’t pay for them! Combined with the simplified deployment flow (compared with applications running in containers or VMs) it has enabled many teams to make use of temporary CloudFormation stacks. In this post, let’s talk about two ways you should use temporary CloudFormation stacks, and why. Disclaimer : this shouldn’t be taken as a prescription. It’s a general approach that has pros and cons, which we will discuss along the way. Temporary stacks for feature branches It’s common for teams to have multiple AWS accounts, one for each environment. While there doesn’t seem to be a consensus on how to use these environments, I tend to follow these conventions: is shared by the team, this is where the latest development changes are deployed and tested end-to-end. This environment is unstable by nature, and shouldn’t be used by other teams. dev is where other teams can integrate with your team’s work. This environment should be fairly stable so to not slow down other teams. test should closely resemble production, and would often contain dumps of production data. This is where you can stress test your release candidate in a production-like environment. staging and then there’s production Having one account per environment is considered best practice. Some of my AWS friends would go even further and have one AWS account per developer. Personally, I have never felt the need to have one account per developer. After all, there is some overhead for having an AWS account. Instead, I usually settle for one AWS account per team per environment. However, for a particular project, I would often have a separate deployment CloudFormation stack per developer (but in the same dev account). This is especially useful when I’m working on feature branches. When I start work on a new feature, I’m still feeling my way towards the best solution for the problem. The codebase is still unstable and many bugs haven’t been ironed out yet. Deploying my half-baked changes to the dev environment can be quite disruptive: it risks destabilising the team’s shared environment it’d overwrite other features the team is working on team members would be fighting over (not literally!) who gets to deploy their feature branch to the shared environment Some of these challenges can be mitigated with feature toggles. Different feature branches can be live at the same time and only enabled for the developers working on them. Services like LaunchDarkly is great for this, even if using it with Lambda . requires extra work But what do you do when you need to deploy and test some unfinished changes? That is, the change is not complete and not yet ready to be reviewed and merged back into . Feature toggles won’t help you in this case. master Instead, I can deploy the feature branch to a dedicated environment, e.g. . Using the , that is as easy as running the command . This would deploy all the Lambda functions, API Gateway and any other related resources (DynamoDB, etc.) in its own CloudFormation stack. I would be able to test my work-in-progress feature in a live AWS environment. dev-my-feature Serverless framework sls deploy -s dev-my-feature Having these temporary CloudFormation stacks for each feature branch have negligible cost overhead. There is no traffic in the account since it’s only used by the team. When the developer is done with the feature, the temporary stack can be easily removed by running . dev sls remove -s dev-my-feature However, since these temporary stacks are an extension of your feature branch. They exhibit the same problems when you have long-lived feature branches. Namely, they get out-of-synch with other systems they need to integrate with. Both in terms of the events coming into your function, such as the payloads from SQS/SNS/Kinesis, etc. As well as data your function depends on, such as those that reside in DynamoDB. While it’s not a problem with serverless technologies per se, but I find teams move faster when they use serverless. Which means the problems with long-lived feature branches become more prominent and noticeable as well. Don’t leave feature branches hanging around for more than, say, a week. If the work is large and takes longer to implement, then break it up into smaller features. And when you’re working on a feature branch, integrate from master regularly (no less than once per day). running localstack vs talking to AWS Instead of spending lots of time to get tools such as working, I find it much more productive to deploy a temporary CloudFormation stack in AWS and run against the real thing. localstack The main downsides are: you need an internet connection deploying to AWS is slower than running code locally, which slows down the feedback loop The internet access argument is only relevant for a handful of people who spend most of their time on the road. I travel more than most and do a lot of work while I’m at airports, and internet access is rarely a problem for me. As for the slower feedback loop, it probably feels worse than it actually is. Most of my deployments are less than 30 seconds. But they do feel like an eternity when I’m staring at the screen and waiting for it to be done. To compensate for the loss of feedback loop, I also use tests as well as to run my functions locally while talking to the real AWS services. sls invoke local Temporary stacks for end-to-end tests Speaking of testing, another common use of temporary CloudFormation stacks is for running end-to-end tests. One of the common problems with these tests is that you need to insert test data into a live, shared AWS environment. As a rule of thumb I always: insert the data a test case needs before the test delete the data after the test finishes These help keep my tests robust and self-contained as they don’t implicitly rely on data to exist. They also help reduce the amount of junk that is floating around in the shared environment too. dev However, despite our best intentions, mistakes happen and sometimes we deliberately cut corners to gain agility in the short-term. Over time, these shared environments are filled with test data, which at times interfere with normal operations. As a counter-measure, many teams would employ cron jobs to wipe these environments from time-to-time. An emerging practice to combat these challenges is to create a temporary CloudFormation stack during the CI/CD pipeline instead. The temporary stack is used to execute the end-to-end tests and destroyed afterwards. This way, there is no need to clean up test data, either as part of your test fixture or with cron jobs. The drawbacks are: the CI/CD pipeline takes longer to run you still leave test data in external systems, so it’s not a complete solution You should weigh the benefits of this approach against the delay it adds to your CI/CD pipeline and decide if it’s right for a project. Personally, I think it’s a great approach and I’d encourage more teams to adopt it. However, the more data you have to clean up in external systems (that is, systems that are not provisioned as part of the temporary stack) the less useful it becomes. Summary In summary, here are two ways you can use temporary CloudFormation stacks to improve your development flow for serverless applications: use temporary stacks for feature branches to avoid destabilising shared environments use temporary stacks during CI/CD pipeline to run end-to-end tests to remove the overhead of cleaning up test data These approaches should not be considered as a prescription. You need to consider their pros and cons and see they fit with your constraints and how your teams work. Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here Originally published at https://theburningmonk.com on September 12, 2019.

Flow

Stacks

LaunchDarkly

Twitter

How to include SNS and Kinesis in your e2e tests

Serverless Step Functions: Avoid Leaky Abstractions

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

A Case for Using Temporary Stacks when you do Serverless

A Case for Using Temporary Stacks when you do Serverless

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps