Making Terraform and Serverless framework work together

The Serverless framework is the most popular deployment framework for serverless applications. It gives you a convenient abstraction over CloudFormation and some best practices out-of-the-box:

• Filters out dev dependencies for Node.js function.
• Update deployment packages to S3, which lets you work around the default 50MB limit on deployment packages.
• Enforces a consistent naming convention for functions and APIs.

But our serverless applications is not only about Lambda functions. We often have to deal with share resources such as VPCs, SQS queues and RDS databases. For example, you might have a centralised Kinesis stream to capture all applications events in the system. In this case, the stream doesn’t belong to any one project and shouldn’t be tied to their deployment cycles.

You still need to follow the principle of Infrastructure as Code:

• version control changes to these shared resources, and
• ensure they can be deployed in a consistent way to different environments

You can still use the Serverless framework to manage these shared resources. It is an abstraction layer over CloudFormation after all. Even without Lambda functions, you can configure AWS resources using normal CloudFormation syntax in YAML.

But this is often frowned upon by DevOps/infrastructure teams. Perhaps the name “Serverless” makes one assume it’s only for deploying serverless applications. On the other hand, Terraform is immensely popular in the DevOps space and enjoys a cult-like following.

I see many teams use both Terraform and Serverless framework in their stack:

• Serverless framework to deploy Lambda functions and their event sources (API Gateway, etc.).
• Terraform to deploy shared dependencies such as VPCs and RDS databases.

The Serverless framework translates your serverless.yml into a CloudFormation stack during deployment. It also lets you reference outputs from another CloudFormation stack. But there’s no built-in support to reference Terraform state. So there is no easy way to reference the shared resources that are managed by Terraform.

Here at DAZN we have used a simple trick to make Serverless framework and Terraform work together. Reading the Terraform state from the Serverless framework is tricky. So, we cheat ;-)

We would create a CloudFormation stack as part of every Terraform script. This CloudFormation stack would hold the output from the resources that Terraform creates? — ?ARNs, etc. We would then be able to reference them from our serverless.yml files.

Let’s look at a simple example.

Example

Here’s a simple Terraform script that provisions a SQS queue.

To export the ARN and URL of this queue, we need to add a CloudFormation Stack to our script. Notice that the stack specifies the outputs MyQueueArn and MyQueueUrl. This is all we wanted to do here. But unfortunately CloudFormation requires you to specify at least one resource…

Since the stack is here to provide outputs for others to reference, let’s stay with that theme. Let’s expose the SQS attributes as SSM parameters as well.

After you run terraform plan and terraform apply you will be able to find the my-terraform-outputs stack in CloudFormation. You will find the URL and ARN for the SQS queue in this stack’s output.

From here, we can reference these outputs from a serverless.yml file.

Since our stack also created SSM parameters for these outputs, we can also reference them from SSM Parameter Store too.

Alternatives

The Serverless framework lets you reference variables from a number of AWS services:

• Another CloudFormation stack’s output.
• A JSON file in S3.
• SSM Parameter Store.
• Secrets Manager.

So you don’t have to use CloudFormation as a way to store outputs from Terraform. Which as you can see, forces you to also provision some resources via CloudFormation…

Assuming we’re not talking about application secrets (which, is a whole separate topic) you should consider outputting them to SSM parameters instead.

Hi, my name is Yan Cui. I’m an AWS Serverless Hero and the author of Production-Ready Serverless. I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless.

You can contact me via Email, Twitter and LinkedIn.

Check out my new course, Complete Guide to AWS Step Functions.

In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices.

Get your copy here.

Come learn about operational BEST PRACTICES for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more.

You can also get 40% off the face price with the code ytcui.

Get your copy here.