You may not realise that you can write AWS Lambda functions in a recursive manner to perform long-running tasks. Here’s two tips to help you do it right. AWS Lambda the maximum execution time of a single invocation to . Whilst this limit might be raised in the future, it’s likely that you’ll still have to consider timeouts for any long-running tasks. For this reason, I personally think it’s a good thing that the current limit is too low for many long running tasks — it forces you to consider edge cases early and avoid the trap of thinking “it be long enough to do X” without considering possible failure modes. limits 5 minutes should Instead, you should write Lambda functions that perform long-running tasks as — eg. . recursive functions processing a large S3 file _we built a system that integrates with BigQuery results and is capable of sending millions of push notifications in a…_hackernoon.com Yubl’s road to Serverless — Part 4, Building a scalable push notification system Here’s 2 tips to help you do it right. use context**.**getRemainingTimeInMillis() When your function is invoked, the object allows you to find out how much time is left in the current invocation. context _While a Lambda function is executing, it can interact with AWS Lambda to get useful runtime information such as:_docs.aws.amazon.com The Context Object (Node.js) - AWS Lambda Suppose you have an expensive task that can be broken into small tasks that can be processed in batches. At the end of each batch, use to check if there’s still enough time to keep processing. Otherwise, and pass along the current position so the next invocation can continue from where it left off. context.getRemainingTimeInMillis() recurse use local state for optimization Whilst Lambda functions are ephemeral by design, containers are still reused for optimization which means you can still leverage in-memory states that are persisted through invocations. You should use this opportunity to avoid loading the same data on each recursion — eg. you could be processing a large S3 file and it’s more efficient (and cheaper) to cache the content of the S3 file. I notice that AWS has also updated their Lambda page to advise you to take advantage of container reuse: best practices However, as Lambda can recycle the container between recursions, it’s possible for you to lose the cached state from one invocation to another. Therefore, you shouldn’t assume the cached state to always be available during a recursion, and always check if there’s cached state first. Also, when dealing with S3 objects, you need to protect yourself against content changes — ie. S3 object is replaced, but container instance is still reused so the cache data is still available. When you call S3’s operation, you should set the optional parameter with the of the cached data. GetObject If-None-Match ETag Here’s how you can apply this technique. Have a look at this Lambda function that recursively processes a S3 file, using the approach outlined in this post. example _lambda-recursive-s3-demo - Recursive AWS Lambda function for processing large S3 file_github.com theburningmonk/lambda-recursive-s3-demo Like what you’re reading but want more help? I’m happy to offer my services as an and help you with your serverless project — architecture reviews, code reviews, building proof-of-concepts, or offer advice on leading practices and tools. independent consultant I’m based in and currently the only UK-based . I have nearly of with running production workloads in AWS at scale. I operate predominantly in the UK but I’m open to travelling for engagements that are longer than a week. To see how we might be able to work together, tell me more about the problems you are trying to solve . London, UK AWS Serverless Hero 10 years experience here I can also run an to help you get with your serverless architecture. You can find out more about the two-day workshop , which takes you from the basics of AWS Lambda all the way through to common operational patterns for log aggregation, distribution tracing and security best practices. in-house workshops production-ready here If you prefer to study at your own pace, then you can also find all the same content of the workshop as a I have produced for Manning. We will cover topics including: video course authentication authorization with API Gateway Cognito & & testing running functions locally & CI/CD log aggregation monitoring best practices distributed tracing with X-Ray tracking correlation IDs performance cost optimization & error handling config management canary deployment VPC security leading practices for Lambda, Kinesis, and API Gateway You can also get the face price with the code . Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP). 40% off ytcui

Amazon

Applying the pub-sub and push-pull messaging patterns with AWS Lambda

Write recursive AWS Lambda functions the right way

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps