Many clients have asked me “ ”. how do I record custom metrics from Lambda? Generally speaking, you can either: Publish custom metrics – e.g. send them at the end of an invocation. synchronously Publish custom metrics by writing them to stdout first and then extracting them from CloudWatch Logs. asynchronously Problems with sending custom metrics synchronously The synchronous approach adds latency to invocations. This can be especially problematic when those extra milliseconds are experienced by our users. For example, if the user is waiting for an API response. Individually, the delay might be negligible. CloudWatch metrics typically respond within tens of milliseconds. That is acceptable to most. But they can quickly compound when functions call one another via API Gateway. Moreover, services are most fragile around their integration points – i.e. when they make network calls to other services. Publishing metrics to CloudWatch introduces another integration point that you need to harden. If CloudWatch experiences an outage, surely you would still want your system to stay up, right? Similarly, if CloudWatch experiences elevated response time then you wouldn’t want your functions to timeout as a result! Hence why I generally prefer to record custom metrics asynchronously, even though this approach also has its drawbacks: There is an additional delay in seeing the most recent metric data. It’s sending more data to CloudWatch Logs, which has a cost implication It introduces complexity because you need something to process the logs and turn them into metrics. Sending custom metrics asynchronously In simple cases, where you have few custom metrics, are the way to go. However, this approach does not scale with complexity – when you have lots of functions and custom metrics. CloudWatch metric filters Instead, you can use a Lambda function. To make it really easy for you to record custom metrics asynchronously, I have published a new application to the . You can also check out the source code on GitHub . Serverless Application Repository here Getting started You can deploy the app via the AWS console , by clicking the Deploy button and follow the instructions. here Or you can deploy it as part of a CloudFormation stack with AWS SAM: You can do the same via CloudFormation or the Serverless framework. You need to first add the following Transform though. For more details on how to do this with the Serverless framework, read this . post Transform: AWS::Serverless-2016-10-31 We announced this new app yesterday. You can go to the 21:00 mark and see how you can configure everything via CloudFormation, including subscribing all CloudWatch log groups to a Kinesis stream first, before subscribing this app to that stream. live on Twitch Recording custom metrics Once deployed, you would be able to record custom metrics by writing to stdout in this format: MONITORING|<value>|<unit>|<metric_name>|<namespace>|<dimensions> where: value is a float unit can be any of the allowed CloudWatch metric units namespace: custom metrics would appear under this namespace, e.g. dimensions: comma-separated key-value pairs, e.g. service=content-item,region=eu-west-1 These messages would be processed and published as custom metrics in CloudWatch metrics. All without adding latency to your invocations! Future works I hope you enjoy this new app, and please feel free to suggest improvements via GitHub . Here are some ideas I have for making it more useful: issues Parse the REPORT messages at the end of an invocation and turn Billed Duration, Memory Size and Memory Used into metrics. Since AppSync doesn’t report resolver metrics to CloudWatch, we can parse the and report resolver duration as metrics. resolver logs Hi, my name is . I’m an and the author of . I have run production workload at scale in AWS for nearly 10 years and I have been an architect or principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. I currently work as an independent consultant focused on AWS and serverless. Yan Cui AWS Serverless Hero Production-Ready Serverless You can contact me via , and . Email Twitter LinkedIn Check out my new course, . Complete Guide to AWS Step Functions In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices. Get your copy . here Come learn about operational for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more. BEST PRACTICES You can also get off the face price with the code . 40% ytcui Get your copy . here Originally published at https://theburningmonk.com on July 25, 2019.

Twitter

A simple Event-Sourcing Example Using Lambda and DynamoDB [Includes Snapshots]

AWS Lambda: how to share code in a monorepo

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Introducing a better way to record custom metrics

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Modeling Hierarchical Access With AppSync In 3 Simple Steps

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Modeling Hierarchical Access With AppSync In 3 Simple Steps

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps