How to control asynchronous invocations of your AWS Lambda functions

Photo by Markus Spiske on Unsplash

The Problem

I have a serverless process that runs in a row — one Lambda runs and before it ends, it invokes another Lambda. Due to rate limits and other errors, I needed to know if the invocation was successful, but I didn’t want to wait until the new Lambda ended. This was challenging.

The other annoying thing was that the number of running Lambdas was increasing out of control, even though I should have one Lambda alive at a time (maybe two, because I start a new Lambda and only then close the current one — an overlap of few milliseconds).

The Solution

There are two types of lambda invocations: Sync (“RequestResponse”) and Async (“Event”).

Initially, I started with the Async option as I didn’t want to wait for the new Lambda to end. It turns out that AWS adds this invocation request to a queue and you have no idea whether it was invoked successfully. Furthermore, if the invocation failed then AWS may re-run it again..

My next attempt was with the Sync invocation. In order to know if the Lambda invocation succeeded, I needed to wait for the new Lambda to end. So I used a little trick:

1. Invoke Sync Lambda
2. Wait for a second
3. After one second, of three things happened:
4. I got a successful invocation from AWS within a second (meaning that the new lambda ran for less than a second — not likely to happen)
5. I got an error from AWS
6. A timeout occurred, so I assume that invocation was successful. The new Lambda is running, but I no longer wait for it.

Important notes from my experience:

1. If a timeout occurred, I assumed that invoke was successful and continued with my code BUT at some point in time the new function will end and reply with a real success or failure response. You must prepare your code to handle it. You may get two flows instead of one — watch out!
2. Same scenario as the previous, but this time the first Lambda is over before the invoked one replies with an answer — AWS will consider the invocation as failed and re-run the Lambda!
3. Lambda socket timeout: If Lambda A invokes a new Lambda B in a synchronous way (“RequestResponse”), it will wait for it to end according to the Lambda socket timeout (lambda.Config.httpOptions = {timeout: 1500};) — even if your code is over and you replied with 200 or 500 response!
4. You can force Lambda to end and not wait for the socket timeout with: “context.callbackWaitsForEmptyEventLoop = false;” — note that you will get an error message when the invoked lambda ends (B) and it will re-invoke in this case too.

Conclusion

• I wanted to know whether Lambda invoked successfully, but I could not find a straightforward solution. You need to use a Sync invocation and “hack” the system.
• Be aware of additional invocations when AWS considers your Lambda as failed just because you didn’t wait for its response.
• Your Lambda function will be invoked multiple times — your code should be idempotent — code logic should know how to handle multiple invocation with the same input (event).

Other solutions that not discussed in this post:

• Lambda Dead Letter Queue — where a failed lambda is inserted to a queue that needs to be managed, the retry invocation on error is still happening in this case.
• Use an external DB to manage your invocation. For example, increment a dynamo counter when Lambda starts and decrease it when it ends.

Amos Shahar is Director of DevOps & IT at superQuery. Via our SQL IDE for Google BigQuery, superQuery uses AI and smart caching to enable data teams to run the most efficient and cost-effective queries on Google BigQuery.