How We Streamlined Serverless Testing by@cwidanage

How We Streamlined Serverless Testing

April 12th 2018 1,192 reads
Read on Terminal Reader
react to story with heart
react to story with light
react to story with boat
react to story with money
Chathura Widanage HackerNoon profile picture

Chathura Widanage


If this is not the first time you hear about Serverless and AWS Lambda, you might have already seen the Orange Test button in AWS Lambda console which becomes useless when you want to use 3rd party dependencies in your serverless project or if you upload a package of the size larger than a certain size limit.


AWS Lambda Test Button


Okay. 🙄 Thanks for the information!

Since this completely breaks the rhythm of the developer, for Sigma we built a test button, which looks & behaves a little bit different.


Sigma’s Test button

This button,

  • Works Faster!
  • Doesn’t become useless when you want to add dependencies.
  • Doesn’t become useless when your lambda code is larger.
  • Tries to avoid cold starts as much as possible for code changes & run-time exceptions.
  • Allows you to manage Test Events more efficiently.


Testing Your Code in Sigma

We made all of above possible by utilizing the base concept of the simple trick that I have mentioned in,

Writing a self-sufficient AWS Lambda Function_If you have already gone through the SLAppForge’s news articles and blogs, you might already know that bunch of…

Deploying the Testing Environment on AWS

To achieve all the above-mentioned features, Sigma creates a Test environment on AWS when user signup for the first time. This test environment mainly has three major components.

  1. Dependency Downloader Lambda function


Dependency Downloader

This lambda is responsible for handling all node dependency updates. I’ll explain about the functionality of this function in detail, later in this story.

2. Test Runner Lambda


Test Runner Lambda

This lambda will be directly invoked by Sigma IDE to execute the test events on project lambda files.

3. Code Bucket


Code Bucket

This bucket will be directly utilized by Dependency Downloader Lambda to persist the downloaded dependencies and by Sigma IDE’s Testing Service to persist source files of the project.

Handling Dependency Updates


Dependency Update Flow

Unlike in other serverless frameworks, adding a dependency is not a painful process in Sigma.


Adding a dependency in Sigma

Once you add a dependency to the project, Sigma sends this update in the background to the Dependency Downloader function which is already deployed in your own AWS account.

The payload of the Dependency Downloader request takes the following format.

{"project": "${projectName}","bucketName":"${bucketName}","dependencies": {"dependency1": "version1","dependency2": "version2","dependency3": "version3",}}

Dependency Downloader creates a package.json file inside the /tmp/{project_id} directory of its container using the dependencies property of above event payload.

With every lambda container(every instance of your function), you get a writable non-persistent disk capacity of 512MB at /tmp directory.

Then following npm command with prefix flag is executed to download dependencies to /tmp/{projectName}/node_modules directory

npm --prefix /tmp/projectName install /tmp/projectName

Since this lambda has nothing else to do concurrently, we use syncExec function to spawn all child processes.

Once syncExec returns back to the parent process, archiver is used to generate a zip of freshly created node_modules directory.

Generated archive (, is then uploaded to the S3 bucket(Code Bucket) in your AWS account with the key {projectName}/

What is a key in S3?

When you create an object, you specify the key name, which uniquely identifies the object in the bucket. Read more.

Testing project function Code


When user click on the ▶️ button, following things happen in order.

  1. Calling Invoke Function command

Testing Service of Sigma receives the payload of the test event, and the name of the lambda function to invoke.

2/3. Check source code diff & Update the diff in S3

Test Service periodically checks for the source javascript files in the Sigma editor for updates. If an updated file is found, it flags that file as updated and wait for any other subsequent updates. If it doesn’t see any succeeding updates for a while, the content of that particular file is pushed to the S3(Code Bucket).

Apart from that, when a test event is invoked, it immediately checks all JS files in the project for updates and pushes them immediately to the S3 if there are any updates. Due to the periodical update checking that I have mentioned above, number of updated files that will be detected at Test invocation time will be 1 in most of the cases.

Source files will be uploaded to the S3 with key {projectName}/{fileName}. So this will essentially create a mirror of the project file structure in S3.

4. Invoke Test Runner

At this step, Testing Service sends Test Event & the name of the Lambda file to be invoked to the Test Runner Lambda. The payload of this invocation takes the following format.

{"project": "${projectName}","type": "RUN","lambda": "${path of file relative to the project root}","bucketName":"${bucketName}","testEvent": ${event}}

5. Checking for code updates

Test Runner function checks for the code updates by checking the ETag of S3 objects with the prefix {projectName}.

What is ETag?

In AWS S3, The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. The ETag may or may not be an MD5 digest of the object data.

Source files with updated content are then downloaded to the /tmp/{projectName} directory of the lambda container.

With each lambda instance, you get a non-persistent space of 512MB at /tmp location of the lambda container.

At the same time; Test Runner generates(or updates) a collection of {FileName:ETag} tuples and assign it to a global variable declared in the Test Runner function. Since values of global variables are preserved as long as lambda container is running, these values will be available in the next test invocation as well. As a result, number of files that should be downloaded in the upcoming invocations will be minimized.

At the end of this step, Test Runner will have an exact similar copy of project files structure in the /tmp/{projectName} directory of the container.

6. Checking for dependency updates

We already have the dependency bundle for our project at {s3Bucket}/{projectName}/

Test Runner applies the same procedure explained above to check whether has any updates since the last invocation.

If it finds any mismatch in Etag of, Test Runner downloads and extracts it to the /tmp/{projectName} of lambda container.

Unlike in Dependency Downloader function, Step 5 & 6 above happen concurrently, and all S3 GET actions are also executed concurrently.

7. Invoking the lambda function

At the end of above 6 steps, inside the /tmp/{projectName} directory of lambda container, we have a fully functioning node.js project with all required dependencies.

Test Runner Lambda now invokes the intended lambda handler as follows.

It is essential to wrap above handler call with a try/catch block to preserve Test Runner lambda container instance in case of a run-time exception.

Apart from the above-mentioned event, which is of type RUN, Test Runner lambda supports another event type, WARM_UP.

{"project": "${projectName}","type": "WARM_UP","bucketName":"${bucketName}","testEvent": {}}

Sigma periodically sends WARM_UP events to the Test Runner lambda_,_ in order to keep it’s container warmer. Apart from keeping the container warmer, on this event; Test Runner performs 5th and 6th steps above in order to keep /tmp directory up to date with the project.

Even after performing a dependency update, Sigma sends a WARM_UP request to Test Runner, which makes RUN events faster by keeping the container ready with dependencies.

In most of the test invocations, Test Runner lambda can skip 6th step above due to the hard work done by WARM_UP requests.

Sigma, at the application level, avoids parallel Test Runner invocations in order to prevent AWS from spawning multiple instances of Test Runner. This makes each subsequent test invocations extremely fast. However this is not a guaranteed behavior. If AWS spawn an additional container by accidental concurrent invocation, Sigma will experience a cold start delay just for a single test invocation. But this happens very rarely.

So that’s how we streamlined Serverless testing & if you came this far with me, you might be wondering what Sigma is. So it is the best time to learn about Sigma!!

What is Sigma?

Sigma was born on 6th of February 2018 in Sri Lanka at SLAppforge, and it was designed & developed with the vision of making the lives of Serverless developers easier.

If you are very new to Sigma, let me give you a brief introduction.

Sigma is a tool which makes Serverless development smoother.

  • Sigma is not an old-fashioned command line tool.
  • You don’t have to install Sigma on your computer.
  • Sigma runs entirely on your browser(Since chrome is not the only browser in the world, we support all modern web browsers).
  • Sigma gives you the rich experience of a modern integrated development environment (IDE).
  • Sigma updates automatically.


You won’t see this in Sigma.

  • Sigma generates code(API calls for other serverless services) for you. So developers don’t have to bother about reading API docs.
  • Sigma manages resources (Resources = DynamoDB tables, S3 buckets, etc in AWS) for you.

When it comes to serverless development, Sigma gives everything you need to deploy a serverless application from the scratch. Our primary goal is to let you develop your app from a-z within the same browser tab.

So that’s about Sigma and since we have been talking about Sigma’s test framework, let me introduce you the other components of the Test Framework which makes your life easier.

Components of Sigma Test framework

Test Event Manager


Sigma Test Event Manager

Sigma’s Test Event Manager gives you the ability to,

  • Define event payloads for individual functions in your project
  • Duplicate an existing event to create a new test event for the same function or for a different function
  • Invoke functions directly with a desired test event



Sigma Trail

SigmaTrail displays all your Test invocation results in real time. Apart from the [TEST] logs, SigmaTrail also displays [Prod] logs for each function which will be generated in production environment when your functions are actually deployed on AWS. Thanks to the SigmaTrail, no more checking CloudWatch for your serverless logs.

Hope you enjoyed! Happy coding Serverless with Sigma.

Call To Action

  • Clap. Appreciate and let others find this article.
  • Comment. Share your views on this article.
  • Follow me. Chathura Widanage to receive updates on articles like this.
  • Keep in touch. LinkedIn, Twitter

SLAppForge_SLAppForge brings serverless computing and applications into mainstream use for the business success of end

Serverless - Hacker Noon_Read writing about Serverless in Hacker Noon. how hackers start their

react to story with heart
react to story with light
react to story with boat
react to story with money
. . . comments & more!