The Problem Throughout my career, I had to implement mechanisms for scheduling ad-hoc tasks a number of times. Often, they are part of a much bigger system. For instance: A tournament system for games would need to execute business logic when the tournament starts and finishes. An event system (think or ) would need a mechanism to send out timely reminders to attendees. eventbrite.com meetup.com A to-do tracker (think ) would need a mechanism to send out reminders when a to-do task is due. wunderlist The TL;DR is that, I need a way to execute a piece of code at a specified point in time in the future. It’s possible to do this in just about every programming language. For example, .Net has the class and JavaScript has the function. But I find myself wanting a service abstraction to work with instead. Sadly, AWS does not offer a service for this type of workloads. is the closest thing, but not quite. [Timer](https://docs.microsoft.com/en-us/dotnet/api/system.timers.timer?view=netframework-4.7.2) [setInterval](https://www.w3schools.com/jsref/met_win_setinterval.asp) CloudWatch Events What is CloudWatch Events? CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. EC2 instance terminated, ECS task started, Lambda function created, etc. To react to these system events, you can subscribe a Lambda function to an Event Pattern. Whenever an event is matched, CloudWatch Events would invoke the target Lambda function on your behalf. In addition, CloudWatch Events also lets you create cron jobs easily. This lets you invoke a Lambda function based on a fix rate (down to every minute). Or, you can specify a custom schedule using a cron expression. The problem with CloudWatch Events as a scheduling mechanism However, CloudWatch Events is not designed for running lots of ad-hoc tasks, each to be executed once, at a specific time. The default limit on CloudWatch Events is a lowly . It’s a soft limit, so it’s possible to request a limit increase. But the low initial limit suggests it’s not designed for use cases where you need to schedule . 100 rules per region per account millions of ad-hoc tasks CloudWatch Events is designed for executing recurring tasks. Because there are no other suitable AWS services, I had to implement a scheduling service myself a few times in my career. I experimented with a number of different approaches: cron job (with CloudWatch Events) wrapping the .Net class as an HTTP endpoint Timer using SQS Visibility Timeout to hide tasks until they’re due Lately, I have seen a number folks use (TTL) to implement these ad-hoc tasks. In this post, we will take a look at this approach and see where it might be applicable for you. DynamoDB Time-To-Live How do we measure the approach? For this type of ad-hoc tasks, we normally care about: : how close to my scheduled time is the task executed? The closer the better. Precision : can the solution scale to support many open tasks. I.e. tasks that are scheduled but not yet executed. Scale (number of open tasks) can the solution scale to execute many tasks around the same time? E.g. millions of people set timer to remind themselves to watch the superbowl, so all the timers fire within close proximity to kickoff time. Scale (hotspots): DynamoDB TTL as a scheduling mechanism From a high level this approach looks like this: A DynamoDB table which holds all the tasks that are scheduled for execution. scheduled_items A function that writes the scheduled task into the table, with the TTL set to the scheduled execution time. scheduler scheduled_items A function that subscribes to the DynamoDB Stream for and react to events. These events corresponds to when items have been deleted from the table. execute-on-schedule scheduled_items REMOVE Scalability (number of open tasks) Since the number of open tasks just translates to the number of items in the table, this approach can scale to millions of open tasks. scheduled_items DynamoDB can handle large throughputs (thousands of TPS) too. So this approach can also be applied to scenarios where thousands of items are scheduled per second. Scalability (hotspots) When many items are deleted at the same time, they are simply queued in the DynamoDB Stream. AWS also autoscales the number of shards in the stream, so as throughput increases the number of shards would go up accordingly. But, events are processed in sequence. So it can take some time for your function to process the event depending on: its position in the stream, and how long it takes to process each event. So, while this approach can scale to support many tasks all expiring at the same time, it cannot guarantee that tasks are executed on time. Precision This is the big question about this approach. According to the official , expired items are deleted . That is a huge margin of error! documentation within 48 hours As an experiment, I set up a Step Functions state machine to: add a configurable number of items to the table, with TTL expiring between 1 and 10 mins scheduled_items track the time the task is scheduled for and when it’s actually picked up by the function execute-on-schedule wait for all the items to be deleted The state machine looks like this: I performed several runs of tests. The results are consistent regardless the number of items in the table. A quick glimpse at the table tells you that, on average, a task is executed over AFTER its scheduled time. 11 mins I repeated the experiments in several other AWS regions: I don’t know why there are such marked difference between US-EAST-1 and the other regions. One explanation is that the TTL process requires a bit of time to kick in after a table is created. Since I was developing against the US-EAST-1 region initially, its TTL process has been “warmed” compared to the other regions. Based on the result of my experiment, it will appear that using DynamoDB TTL as a scheduling mechanism cannot guarantee a reasonable precision. On the one hand, the approach scales very well. But on the other, the scheduled tasks are executed at least several minutes behind, which renders it unsuitable for many use cases. Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here

Amazon

Microsoft

Target

Timely

Twitter

Understanding the scaling behaviour of DynamoDB OnDemand tables

Chaos test your Lambda functions with Thundra

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

DynamoDB TTL as an ad-hoc scheduling mechanism

DynamoDB TTL as an ad-hoc scheduling mechanism

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Modeling Hierarchical Access With AppSync In 3 Simple Steps

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Modeling Hierarchical Access With AppSync In 3 Simple Steps

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps