Adding Code to AWS Lambda, Lambda Layers, and Lambda Extensions Using Docker

2020 was a difficult year for all of us, and it was no different for engineering teams. Many software releases were postponed, and the industry slowed its development speed quite a bit.

But at least at AWS, some teams released updates out of the door at the end of the year. AWS Lambda received two significant improvements:

• AWS Lambda Extensions; and
• Support of Docker images for your functions.

With these two new features and Lambda Layers, we now have three ways to add code to Lambda that isn’t directly part of our Lambda function.

The question is now: when should we use what?

In this article, I try to shine some light on the Lambda Layers, Lambda Extensions, and Docker image for Lambda.

First things first. All these Lambda features can be used together. So if you think about where to put your code, at least your decisions aren’t mutually exclusive. You can upload a Docker image and attach a regular Lambda Layer and a Lambda Extension. The same is possible if your Lambda function is based on a ZIP archive.

What does this all mean? Keep reading and find out.

Lambda Functions with Docker

Let’s look at Lambda functions based on Docker images. Many developers adopted Docker containers into their development environment because it helps replicate the production environment as close as possible. After all, many services today are running on Kubernetes, which manages Docker containers.

AWS Lambda is a different approach to microservices. It’s not based on long-running containers but short-lived functions. It originally came with its own way of doing development, and at first, people couldn’t use their Docker skills. But since re:invent 2020, AWS Lambda now supports Docker images to cater to developers who are already used to a Docker-based development workflow.

Instead of packing your function code into a ZIP archive, you can also pack it into a Docker image. In the opinion of many serverless advocates, this is just seen as a way to pull legacy developers from the “container world” to serverless. Still, it also has a crucial benefit: you get 10GB of space.

That’s right, your Docker images for Lambda can be significantly larger than your ZIP archives, which only allow for 250MB of unzipped code and assets. Sometimes this is just how things are; you pull in a huge dependency and it’s over 250MB in no time. This is also an important factor for legacy systems. When you have a Docker image that you used for many years now, it can very well be that size wasn’t much of a concern for you in the past, so getting all that code down to 250MB just isn’t feasible as it would require you to do a significant refactor into many functions.

So if you have a specific use case that requires more than 250MB, you just won’t get around Docker images for your Lambda function. When your whole image is already based on other Docker images that keep track of monitoring and other things not directly related to your function code, you’re also better off using Docker for your Lambda function instead of trying to refactor everything to fit into Lambda Layers or Extensions.

One note, though: your container image has to implement the Lambda runtime API. You can’t run any Docker image on Lambda; it has to get Lambda events via the runtime API. Your containers won’t run indefinitely; like ZIP-based Lambda functions, they will be paused and subjected to cold-starts. 15 minutes is the maximum allowed time for a function.

Lambda Layers

Layers are helpful in multiple ways. The most prevalent way layers are used is for custom runtimes. This means you can create a layer for a programming language that isn’t officially supported for AWS. There are even runtime layers with community support, so if you’re lucky, you can just pull an open-source runtime and start writing code in the programming language of your choice.

The other way layers are used to share code between Lambda functions. The code of a layer will be available when your function runs, so you can import it as if it were part of your function, but since it is encapsulated in a layer, you can maintain it in a separate place. If you have a library, a framework, or just a utils folder with code that gets used all over your Lambda functions, you could pack it into a layer.

While layers can be used together with Docker images, you can see them as separate ways to do the same thing.

If you’re starting a new project, put your functions into ZIP archives and use Lambda layers to factor out code used in multiple places if you can get away with the 250MB limit. This limit is for your whole function with all of its layers, of which every function can have 5. This means you can’t have 5 layers, each 250MB, but the sum of your function code and layers has to be below 250MB.

If you require more than 250MB or simply have a Docker-based legacy code-base that would never fit into 250MB, you’re better off with Docker than with a big refactor, at least in the short term. You can factor your libraries out into Docker images that become the base of all your Lambda functions instead of using Lambda layers and won’t have to fight with the 5 layers per function limit here.

Lambda Extensions

Lambda Extensions are deployed as Lambda Layers; if you want an extension to your Lambda function but can’t find that option in the AWS Console, it’s because an extension is simply a layer.

Extensions are a special layer because they don’t just supply your function with a custom runtime or shared code. Extensions can run in the same process as your Lambda function and in a separate process. This separate process is started before your Lambda function is executed and stopped after it finishes.

Running in a separate process enables Lambda Extensions to do things that regular layers couldn’t normally do. For example, gathering secrets from somewhere before your function does its work, or tracking the time your function runs and gathering errors when it fails. Since an extension runs in its own process, a Lambda function crash usually won’t crash the extension, so it can take its time to find out what went wrong and send it to a monitoring service of your choice.

So if you have code that may need monitoring or observability, or need to perform actions before or after multiple functions run, you should consider putting that code into a Lambda Extension.

Summary

To sum it all up, you should see Docker images mainly as a way to get more code deployed or an easier way to port legacy container code-bases to AWS Lambda. The 250MB of ZIP deployments often won’t cut it, and the 10GB of a Docker image is the only way around it.

Lambda Layers are a way to add libraries, frameworks, util-folders, and custom runtimes to your Lambda function.

Lambda Extensions are Lambda Layers that can run in a separate process, which allows them to perform actions before and after a function executes. Use-cases are monitoring or secret gathering.

With this knowledge, you’re now able to structure your Lambda code in a meaningful way and won’t despair when you hit a code size limit.

Previously published at https://dashbird.io/blog/docker-lambda-layers-extensions/