2,683 reads

What Is Cloudwatch Embedded Metrics?

by JerryNovember 14th, 2022

Too Long; Didn't Read

Cloudwatch Metrics lets you convert your logs into metrics, and when using Lambda it works out of the box!

Companies Mentioned

featured image - What Is Cloudwatch Embedded Metrics?

Goals

After reading the article, you should:

✅ Be able to differentiate between Logs & Metrics
✅ Know the what and why of Cloudwatch Embedded metrics
✅ Gain a high-level understanding of what it takes to convert logs into metrics in AWS

Content

Introduction
What is it?
Why it matters
Supported Platform
Example - using AWS Lambda
Client Libraries
Conclusion

Introduction

If you use AWS, then there is a good chance you are using AWS Cloudwatch logs for your logging.

Logs are very useful for specific events within your code between requests.

However, if you want to gain high-level insights, you want to be looking at an aggregation of events at a point in time for a system rather than a specific event. This is where metrics come in.

Now, the question becomes — how do you go from the logs to the metrics?

If you are using AWS, then there is an easy way to achieve this, and that is using AWS Cloudwatch embedded metrics.

The gist of it is that these are logs formatted in a specific format that AWS uses to convert these logs into metrics.

Let’s go through the details.

What Is It?

AWS Cloudwatch embedded metrics are just logs in a specific format.

It needs to be a valid JSON and contain the right details so AWS can convert these logs into metrics.

Format: x-amzn-logs-format: json/emf

Here is a sample of the format:

{
  "_aws": {
    "Timestamp": 1574109732004,
    "CloudWatchMetrics": [
      {
        "Namespace": "lambda-function-metrics",
        "Dimensions": [["functionVersion"]],
        "Metrics": [
          {
            "Name": "time",
            "Unit": "Milliseconds"
          }
        ]
      }
    ]
  },
  "functionVersion": "$LATEST",
  "time": 100,
  "requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee"
}

📝 Helpful reference:

AWS - Cloudwatch Embedded metrics specification

Why It Matters

Here are a few points of why I think AWS Cloudwatch embedded metrics matter.

Metrics are actionable

You go from logs (specific events) to high-level insights (metrics), and you can’t improve or analyze what you can’t see.

These metrics can be used to define contracts such as SLAs, SLIs, and SLOs.

AWS Cloudwatch embedded metrics make this process that much easier and simpler.

Great Developer Experience (DX)

This feature is an opt-in feature, meaning it’s optional.

When you want to “enable” it, you just format your log in a specific way.

No friction, no extra steps — It simply just works! In my opinion, that’s a big win.

Shifted Operational Responsibility

In this setup, AWS assumes almost all the responsibility for this feature and infrastructure.

This reduces the operational burdens that you or your team is responsible for.

To me, that’s another big win!

Supported Platforms

Using AWS Lambda, you don’t need any further configuration, it just works.

If you are using AWS ECS or EKS, then the cloudwatch agent is required to make it work.

AWS Lambda - Cloudwatch embedded metrics are supported out of the box

AWS ECS and EKS - Requires the CloudWatch agent to send Cloudwatch embedded metrics

Example - Using AWS Lambda

Here is an example of how AWS Cloudwatch embedded metrics work with AWS Lambda. Again, this works out of the box.

The steps:

API gateway receives request
API Gateway invokes Lambda Alias
Lambda processes the event
Lambda logs to cloudwatch logs ⭐️
Logs are automatically converted to Cloudwatch metrics

Step 1-4 are pretty typical in an AWS Lambda setup, the big difference is that when you log in a certain format, AWS will pick that up and convert those logs into metrics.

Step 4 is where the magic happens!

You can format these logs manually or use a client library provided by AWS (recommended).

Client Libraries

To make it easier, AWS has provided client libraries to generate these formatted logs.

It provides an API to help you create formatted logs that result in desired cloudwatch metrics based on the settings you provide.

The client libraries:

Example using the client library with Node.js:

const { createMetricsLogger, Unit } = require("aws-embedded-metrics");

const sendMetrics = async () => {
  const metrics = createMetricsLogger();
  metrics.putDimensions({ Service: "Aggregator" });
  metrics.putMetric("ProcessingLatency", 100, Unit.Milliseconds);
  metrics.setProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8");
  // ...
  await metrics.flush();
};

await sendMetrics();