How do you control the number of concurrent state machine executions that can access a shared resource? Here at , we are migrat­ing from our lega­cy plat­form into a brave new world of and microser­vices. Along the way, we also dis­cov­ered the delights that AWS Step Func­tion has to offer, for exam­ple… DAZN microfron­tends flex­i­ble error han­dling and retry the under­stat­ed abil­i­ty to wait between tasks the abil­i­ty to mix auto­mat­ed steps with that require human interven­tion activ­i­ties In some cas­es, we need to con­trol the num­ber of con­cur­rent state machine exe­cu­tions that can access a shared resource. This might be a busi­ness require­ment. Or it could be due to scal­a­bil­i­ty con­cerns for the shared resource. It might also be a result of the design of our state machine which makes it dif­fi­cult to par­al­lelise. We came up with a few solu­tions that fall into two gen­er­al cat­e­gories: Con­trol the num­ber of exe­cu­tions that you can start Allow con­cur­rent exe­cu­tions to start, but block an exe­cu­tion from enter­ing the crit­i­cal path until it’s able to acquire a (i.e. a sig­nal to proceed) sem­a­phore Control the number of concurrent executions You can con­trol the MAX num­ber of con­cur­rent exe­cu­tions by intro­duc­ing a SQS queue. A Cloud­Watch sched­ule will trig­ger a Lamb­da func­tion to… check how many con­cur­rent exe­cu­tions there are if there are N exe­cu­tions, then we can start MAX-N exe­cu­tions poll SQS for MAX-N mes­sages, and start a new exe­cu­tion for each We’re not using the new here because the pur­pose is to the cre­ation of new exe­cu­tions. Where­as the SQS trig­ger would push tasks to our Lamb­da func­tion eager­ly. SQS trig­ger for Lamb­da slow down Also, you should use a FIFO queue so that tasks are processed in the same order they’re added to the queue. Block execution using semaphores You can use the API to find out how many exe­cu­tions are in the RUNNING state. You can then sort them by and only allow the eldest exe­cu­tions to tran­si­tion to states that access the shared resource. Lis­tEx­e­cu­tions start­Date Take the fol­low­ing state machine for instance. The state invokes the Lambda func­tion and returns a flag. The state then branch­es the flow of this exe­cu­tion based on the flag. Only­One­Shall­RunA­tOne­Time one-shall-pass proceed Shall Pass? proceed :Type: TaskResource: arn:aws:lambda:us-east-1:xxx:function: Next: Shall Pass? :Type: ChoiceChoices:- Variable: # check if this execution should proceedBooleanEquals: Next: Default: WaitToProceed   # otherwise wait and try again later :Type: WaitSeconds: 60Next: OnlyOneShallRunAtOneTime OnlyOneShallRunAtOneTime one-shall-pass Shall Pass? $.proceed true SetWriteThroughputDeltaForScaleUp WaitToProceed The tricky thing here is how to asso­ciate the Lamb­da invo­ca­tion with the corre­spond­ing Step Func­tion exe­cu­tion. Unfor­tu­nate­ly, Step Func­tions does not pass the exe­cu­tion ARN to the Lamb­da func­tion. Instead, we have to pass the exe­cu­tion name as part of the input when we start the exe­cu­tion. const = uuid().replace(/-/g, '_')const input = JSON.stringify({ , bucketName, fileName, mode })   const req = { stateMachineArn, name, input }const resp = await SFN.startExecution(req).promise() name name When the func­tion runs, it can use the exe­cu­tion from the input. It’s then able to match the invo­ca­tion against the exe­cu­tions returned by . one_shall_pass name Lis­tEx­e­cu­tions In this par­tic­u­lar case, only the eldest exe­cu­tion can pro­ceed. All other executions would tran­si­tion to the state. Wait­To­Pro­ceed module.exports.handler = async ( , context) => {const executions = await listRunningExecutions()Log.info(`found ${executions.length} RUNNING executions`) input const oldest = _.sortBy(executions, x => .getTime())[0]Log.info(`the oldest execution is [${oldest.name}]`) x.startDate if ( === ) {return { ...input, proceed: true }} else {return { ...input, proceed: false }}} oldest.name input.name Compare the approaches Let’s com­pare the two approach­es against the fol­low­ing cri­te­ria: . How well does the approach cope as the num­ber of con­cur­rent exe­cu­tions goes up? Scal­a­bil­i­ty . How many mov­ing parts does the approach add? Sim­plic­i­ty . How much extra cost does the approach add? Cost Scalability Approach 2 (block­ing exe­cu­tions) has two prob­lems when you have a large num­ber of con­cur­rent exe­cu­tions. First, you can hit the region­al throt­tling lim­it on the API call. ListExecutions Sec­ond, if you have con­fig­ured time­out on your state machine (and you should!) then they can also time­out. This cre­ates back­pres­sure on the sys­tem. Approach 1 (with SQS) is far more scal­able by com­par­i­son. Queued tasks are not start­ed until they are allowed to start so no back­pres­sure. Only the cron Lamb­da func­tion needs to list exe­cu­tions, so you’re also unlike­ly to reach API lim­its. Simplicity Approach 1 intro­duces new pieces to the infra­struc­ture — SQS, Cloud­Watch sched­ule and Lamb­da. Also, it forces the pro­duc­ers to change as well. With approach 2, a new Lamb­da func­tion is need­ed for the addi­tion­al step, but it’s part of the state machine. Cost Approach 1 intro­duces min­i­mal base­line cost even when there are no exe­cu­tions. How­ev­er, we are talk­ing about cents here… Approach 2 intro­duces addi­tion­al state tran­si­tions, which is around $25 per mil­lion. See the page for more details. Since each exe­cu­tion will incur 3 tran­si­tions per minute whilst it’s blocked, the cost of these tran­si­tions can pile up quick­ly. Step Func­tions pric­ing Conclusions Giv­en the two approach­es we con­sid­ered here, using SQS is by far the more scal­able. It is also more cost effec­tive as the num­ber of con­cur­rent exe­cu­tions goes up. But, you need to man­age addi­tion­al infra­struc­ture and force upstream sys­tems to change. This can impact oth­er teams, and ulti­mate­ly affects your abil­i­ty to deliv­er on time. If you do not expect a high num­ber of exe­cu­tions, then you might be bet­ter off going with the sec­ond approach. Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here

Flow

Amazon

Twitter

Centralised logging for AWS Lambda, REVISED (2018)

Step Functions: how to implement semaphores for state machines

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps