One-shot containers with Serverless

Have you ever had short lived containers like the following use cases:

• Batch and ETL (Extract, Transform & Load) Jobs.
• Database backups and replication.
• Machine Learning algorithms for generation of learning and training models.
• Integration & Sanity tests.
• Web scrapers & crawlers.

And you were wondering how you can deploy your container periodically or in response to an event ? The answer is by using Lambda itself, the idea is by making a Lambda function trigger a deployment of your container from the build server. The following figure illustrates how this process can be implemented:

I have wrote a simple application in Go to simulate a short time process using sleep method:

As Go is a complied language, I have used Docker multi-stage build feature to build a lightweight Docker image with the following Dockerfile:

Next, I have a simple CI/CD workflow in Jenkins, the following is the Jenkinsfile used to build the pipeline:

An example of the pipeline execution is given as follows:

Now, all changes to the application will trigger a new build on Jenkins which will build the new Docker image, push the image to a private registry and deploy the new Docker image to the Swarm cluster:

If you issue the “docker service logs APP_NAME” on one of the cluster managers, your application should be working as expected:

Now our application is ready, let’s make it execute everyday at 8am using a Lambda function. The following is the entrypoint (handler) that will be executed on each invocation of the function:

It uses the Jenkins API to trigger the deployment process job.

Now the function is defined, use the shell script below to create the following:

• Build a deployment package (.zip file).
• Create an IAM role with permissions to push logs to CloudWatch.
• Create a Go based Lambda function from the deployment package.
• Create a CloudWatch Event rule that will be executed everyday at 8am.
• Make the CloudWatch Event invoke the Lambda function.

As a result, a Lambda function will be created as follows:

To test it out, you can invoke it manually either from the Lambda Console or using the following AWS CLI command:

A new deployment should be triggered in Jenkins and your application should be deployed once again:

That’s it, it was a quick example on how you can use Serverless with Containers, you can go further and use Lambda functions to scale out/scale in your services in your Swarm/Kubernetes cluster by using either CloudWatch events for expected scheduled events (Holidays, Black Friday …) or other AWS managed services like API Gateway in response to incoming client requests.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.