How To Cache Secrets Using AWS Lambda Extensions

Build a cache layer for secrets stored in AWS Secrets manager using AWS Lambda extension

What is the AWS Lambda extension?

A month back AWS announced a preview of Lambda Extensions, a new way to easily integrate Lambda with your favorite monitoring, observability, security, and governance tools. Extensions can be published as Lambda layers, there are two types are extension:

Internal extensions → Run as part of the runtime process, in-process with your code. Internal extensions enable use cases such as automatically instrumenting code.
External extensions → Allow you to run separate processes from the runtime but still within the same execution environment as the Lambda function. External extensions can start before the runtime process and can continue after the runtime shuts down. These extensions run as companion processes to Lambda functions.

Usecase

The whole idea of extensions was fascinating to me. So I thought, instead of just using the extensions for just implementing observability patterns, why can't we build a cache layer for secrets using extensions.

As always, I googled my idea to see anyone has already implemented this. A famous company named “Square” has many smarter engineers than me, and they have already published a Lambda extension with a similar idea in GitHub 😟 . To be clear, they did a fantastic job, but like always, there is room for improvement, isn’t it right?

The extension they published was storing the secret in “/tmp” directory of the Lambda, a BIG NO NO from a security perspective, and no cache refresh logic. So I decided to let me build an extension that can take care of all the below use cases:

Prefetch secrets from AWS Secret manager and save it into an in-memory cache.
The extension will read “config.yaml” file in the lambda function to get the list of secrets that needs to be prefetched and cached. This way, the Lambda function controls what needs to be cached, and the extension code can be generic and reused across multiple Lambdas for a similar use case.
Cache validity is specified through an environment variable in Lambda, instead of extension defining it.

Architecture

Here is the high level view of all the components

This extension demonstrates how to host an HTTP server which cache secrets required for the lambda function. The extension does the following:

Hosts a local HTTP server, which will be invoked by lambda function to retrieve the secret instead of directly going to AWS Secret manager
The extension using config.yaml file inside the Lambda to get the list of secrets that needs to get cached in memory
Secrets cache is refreshed based on the value (in minutes) set in the environment variable “CACHE_TIMEOUT”. If no value is specified its defaulted to 10

Note: This extension requires the Node.js 12 runtime to be present in the Lambda execution environment of your function.

Initializing extension and reading secrets from the cache

Below sequence diagram explains the initialization of lambda extension and how lambda function reads cached secrets using HTTP server hosted inside the extension

Code

Deploy

Pre-requisites

Checkout the code from the GitHub repoNodeJS runtime should be installed in the local system, for more information click here

Zip utility needs to be installed in the local systemAWS CLI needs to be installed in the local system, for more information click here
Create new secret in AWS Secret ManagerCreate a new secret using the following command

Note: This command assumes you’ve placed your secret, such as this example JSON text structure {“username”:”anika”,”password”:”aDM4N3*!8TT”}, in a file named mycreds.json.

aws secretsmanager create-secret --name secret_now --secret-string file://mycreds.json

Lambda function

Create a new NodeJS runtime Lambda function with the name “Secrets-Extension-Lambda-Test”

After successfully creation of Lambda, attach “SecretsManagerReadWrite” policy to the IAM role associated with the Lambda function
Replace the code inside “index.js” with the following

Note: Sample of the below code can be found in “example-function/index.js” under the code repo

Note: The code invokes the local HTTP server hosted inside lambda extension to read the secret’s value instead of directly going to AWS Secret’s manager

Create a new file named “config.yaml” under the root directory with the following contents:

Note: Sample of the below yaml file can be found in “example-function/config.yaml” under the code repo. “secret_now” is the name of the secret we have created in the previous step, and we want to be cached by the extension. If you want more secrets to be cached you can keep adding them here

SecretManagers:

1. secrets:

secret_now

Click on “Deploy” button to deploy the latest changes
Create a new environment variable “CACHE_TIMEOUT” and set the value in minutes based on which the cache will be refreshed

Note: If the environment value not found, then cache gets refreshed every 10 minutes

Increase the memory of Lambda to 1200 MB & Timeout to 30 seconds

Deploy extension

Run the following command to deploy the extension and associate the layer to the Lambda function

> chmod +x deploy.sh
> ./deploy.sh

Invoke Lambda function

You can invoke the Lambda function using the following CLI command

aws lambda invoke \
 --function-name "Secrets-Extension-Lambda-Test" \
 --payload '{"payload": "hello"}' /tmp/invoke-result \
 --cli-binary-format raw-in-base64-out \
 --log-type Tail

The function should return "StatusCode": 200.

View logs

Browse to the Amazon CloudWatch Console. Navigate to Logs\Log Groups. Select the log group /aws/lambda/Secrets-Extension-Lambda-Test.

View the log stream to see the runtime log with pattern "Response from cache" followed by username and password stored in mycred.json

Resources

If you like the write-up and found it helpful, give a clap 👏 or leave a comment in the article. Stay home and stay safe 😷