You can now build your serverless workloads in a Docker container with FaaS. Among all the new features and services that AWS announced during the re:Invent 2020, my favorites were definitely the AWS Lambda updates. And there were many! For example, your code execution is no longer rounded up to the nearest 100ms of duration for billing — you are now billed on a per millisecond. On top of that, AWS increased the Lambda’s memory capacity to 10 GB, and correspondingly the CPU capacity up to 6 vCPUs [3]. But today, I want to dig deeper into something even more exciting for me. Namely, from now on, AWS Lambda doesn’t require packaging your code and dependencies to a zip file. Instead, you can now do it with a Docker container image that can be up to 10 GB in size. Personally, I consider this a game-changer for many serverless use cases. And here’s why. The easiest way to package code for deployment is to use a (Docker) container Until recently, the only way of creating a serverless function on AWS was to select your specific language and runtime ( ), then making sure that you install all your custom dependencies inside of your project directory ( ) and finally, compressing all that into a zip package. If your zip file is bigger than 50MB, you would also have to upload the code to S3 and reference it in your function definition. All that is doable. Many developers (me including) used to create their own methods to make it easier, such as using Lambda layers, site-packages from a virtual environment, and building shell scripts for deployment. ex. Python 3.8 or adding site-packages from a Python’s virtual environment to your zip package On the surface, it seems like not much changes — instead of zipping your code, you now define your dependencies inside a Dockerfile. But there is more to it, as defining your runtime environment in a container image gives you much more control over your environment compared to what you get with predefined runtimes and zipping dependencies. : what if you would like to use a specific Python environment that has been reviewed by your company’s security team? Or what if you need some additional OS-level package? With the container image support, you can do that since a in the base image and packages you choose to install. , and the development of FaaS ( ) becomes much easier. A zip file with a predefined runtime environment has its limits Docker container has no restrictions This makes “serverless” accessible to a wider audience Function as a Service In theory, it’s even possible to create custom images for other programming languages, although this requires implementing a custom runtime and is more involved. The interface of AWS Lambda now looks as follows: Note: at the time of writing, only Linux containers are supported. Container image deployment with a simple example Let’s build a simple ETL example. Here is a project structure that we will use: My requirements.txt contains only: pandas==1.1.0. The actual code, demonstrated below, is just a simple ETL example counting exam scores of Harry Potter’s characters, but you can use it as a scaffold for your use case: Now to the fun part: the Dockerfile that will define all our code dependencies so that we don’t need to zip our code!💪🏻 Usually, your base image for Python 3.8 would start with FROM python:3.8 in order to use the official Python image from the Dockerhub. However, to make it usable with AWS Lambda, your base image must include the Lambda Runtime API. To make it easier for us, AWS prepared many base images that we can use, such as the one defined in line 3 in the Dockerfile presented above. You can find all AWS Lambda images in the public ECR repository as well as in the Dockerhub registry: Let’s test our dockerized Lambda function The best part of developing your Lambda functions with a container image is the dev/prod environment parity. You can easily test your code locally with Docker before deploying your code to AWS. Your local containerized environment is identical to the one you will be using later in production. This is possible due to a web-server environment called Lambda Runtime Interface Emulator (RIE) (you can find out more about it here), which has been open-sourced by AWS. This emulator is already baked into all Lambda images (amazon/aws-lambda-*) that you can find on Dockerhub or in the ECR public image repository. Run the following commands from the project directory that contains Dockerfile: Then, in a new terminal window, run: Here is what I’m getting as output: Local execution looks good. Let’s deploy it to AWS. Pushing your container image to ECR & Creating a Lambda function We can now run the following commands to create an ECR repository and push our container image to ECR: Now that our image is deployed, we can use it in our Lambda function: Note that we didn’t have to select the runtime environment since it’s all already defined in our container image. We tested the function from the AWS management console and saw that we got the same result as when tested locally. Monitoring your Dockerized Workloads with Lambda By now, you may be convinced that , and you may want to use it now much more extensively. However, I encourage you to think ahead about observability and approach the serverless workloads with an architect’s foresight. running containerized workloads with AWS Lambda has a myriad of advantages Imagine that you migrated several data pipelines from a container orchestration solution to AWS Lambda. Sure, AWS offers native support for logging and alerting via Amazon CloudWatch. Still, to be completely honest, to set up proper alerting, configure log groups, and set up everything to ensure tracing with X-Ray. Then, we also need to decide on metrics to track and build CloudWatch dashboards to visualize this data. How do you know which of those pipelines failed and why? AWS services for monitoring and observability require some extra work You can considerably improve the developer experience by using tools such as Dashbird, which allows you to to your existing serverless workloads . All you need to do is to create an IAM role that will grant Dashbird cross-account permission to communicate with your AWS resources. Once that’s configured, you can , such as automated alert notifications, visualizations of your metrics, and actionable insights based on the AWS Well-Architected Framework to improve performance, save costs, and enhance the security of your cloud resources. easily add observability without any changes to your code or infrastructure immediately start enjoying all benefits of the platform Actionable insights gathered by leveraging Dashbird. Recap: the benefits of Container Image as opposed to a zip deployment When using a container image rather than a zip package for your serverless function deployments, you’ll get the following benefits: Support for you want ( ), any programming language as long as you use a base image that implements the Lambda Runtime API Ability to easily work with that can be baked into a container image such as additional Python modules or config files, additional dependencies — you can easily move the same jobs to a K8s cluster or any platform supporting containerized applications. In our example, we would have to only change the base image back to python:3.8 and the entry point command to: CMD [“python”, “etl.py”] within the Dockerfile. Flexibility & independence of any platform with a traditional Lambda runtime environment, you use what you got from AWS. In contrast, with a container image, you have many options to customize the environment to your needs. Imagine that you want to use a smaller and more lightweight Python image for performance and cost optimization or an image that has been approved by the security team to meet your companies’ specific compliance requirements. You have much more control over your packaged environment — — a containerized application minimizes any surprises when moving your code from your local machine to the development, testing, or production environment. Your code can run anywhere without side effects. Your code can run anywhere — while container orchestration platforms such as Kubernetes are great, some use cases may be better served from a simple FaaS, for instance, when you want your code to run every time a new file arrived in S3 or when somebody made a request to our API. AWS Lambda is perfect for such use cases. Run event-driven containers Conclusion I’m quite happy about all the new AWS Lambda features. As a huge proponent of containerized applications, . These days, developing self-contained microservices has become easier than ever before due to the existence of so many platforms and services to run containers at scale. And if you want to ensure observability and enterprise-grade monitoring of your serverless containers, Dashbird is a great option to consider: https://dashbird.io/. I prefer that option over zipping the code for a serverless deployment Previously published at https://dashbird.io/blog/deploying-aws-lambda-with-docker/

Amazon

An In-Depth Guide to AWS Lambda Error Handling 

Why the AWS Console Isn’t the Best for Serverless Debugging

Serverless af

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Devops

Nominated for 2022 - HackerNoon Contributor of the Year - Serverless

Nominated for 2022 - HackerNoon Contributor of the Year - Aws

Too Long; Didn't Read

How to Deploy AWS Lambda with Docker Containers

How to Deploy AWS Lambda with Docker Containers

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Things in Engineering We Don't Spend Enough Time On

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

10 Things in Engineering We Don't Spend Enough Time On

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps