For the past two years, I have focused most of my time and energy on building Serverless applications. The culmination? Me and my friend founded Dashbird — a monitoring and error alerting service for AWS Lambda. Before starting Dashbird, we built serverless solutions in Testlio — a crowd sourced QA company. Today, both of these services heavily use Lambda functions.
There are many areas in serverless that I would like to cover, but I’ll focus on the elephant in the room: monitoring and getting insights into Lambda functions. I think it’s one of the biggest problems in the serverless space while also being the area I consider my expertise to be the most impactful.
When adopting serverless, we had to re-imagine our approach on obtaining and displaying application metrics. We wanted to keep our functions clean and simple — no third-party agents and wrappers. We also wanted everything to be observable through a single dashboard — a feature that existing APMs were lacking. And most of all, we wanted the ability to find and drill down into invocation level data, including logs and context, to troubleshoot and debug code when something went haywire.
Today, I can say that we are pretty close to that level of visibility and I want to share my experience and take-aways of how we got there.
Analysing logs is an extremely powerful way of gathering information and there isn’t much you can’t do with it. But with Lambda, you can take this to a whole new level.
Let me explain.
CloudWatch organises logs based on function, version and containers while Lambda adds metadata for each invocation. In addition, runtime and container errors are included in the logs. And of course, you can log out any custom metric and have it turned into time-series graphs. It’s not a job for CloudWatch, though.
Generally speaking, there are two angles for monitoring an application. System metrics (like latency, errors, invocations and memory usage) and business analytics, like the number of signups or emails sent.
Technical performance metrics and error detection is pretty universal and that is what Dashbird is meant to be — a plug and play monitoring service.
Business metrics, however, vary from service to service and need a custom approach. Our weapon of choice for that is SumoLogic, but you can use other services like Logs.io etc.
Let’s tackle system metrics first…
We built Dashbird to get visibility into technical metrics of serverless architectures. It works by collecting and analysing CloudWatch logs in real-time.
With all good monitoring services, it’s important to get an overview on a single screen. The main page is designed to do just that. It includes an overview of all invocations, top active functions, recent errors and system health. It’s supposed to tell you if and where you have problems.
From there, you can go down to Lambda view and analyse each function individually.
This view enables developers to judge latency and memory usage. We use it to optimise functions for cost efficiency by adjusting the provisioned memory to match the actual usage. Alternatively, it’s useful for speeding up endpoints by adding more memory.
For troubleshooting and fixing problems, we rely on the failure recognition in logs. In our experience, this approach is just right for Lambda functions.
Here are some of the reasons:
The story doesn’t end there here. Regardless of all the fancy graphs, I’ve still found myself clueless of what happened more times than I’d like to admit. Merely a stack-trace might not be enough to understand the details of a failed execution (especially with Node.js’s fuzzy traces). For that, we’ve developed some conventions for logging in Lambda functions.
Business analytics follow the same basic idea. Our weapon of choice is SumoLogic.
SumoLogic is a machine data analytics service for log management and time series metrics.
What’s great about the service is the ability to construct custom dashboard out of pretty much anything. The setup is a bit different from Dashbird but it’s just as awesome. Here’s a lambda function that subscribes to a log group and sends logs to the service 😎.
There isn’t as much convention and common ground in custom metrics, so we’re going to play this through with an example.
I’m going to demonstrate how we gathered metrics for an integration service. The service has the task on syncing issues between issue-tracker accounts (think JIRA and Asana). We wanted to log out all CRUD actions against client issue-trackers.
For that, let’s add a log line for each time a request of this sort is made:
Now we have the ability to turn these events into time-series metrics. Let’s query this…
"-metrics.integrations.prod.crud." | parse "-metrics.integrations.prod.crud.*" as method | timeslice 5m | count(method) group by _timeslice, method | transpose row _timeslice column method
and see what we get…
Nice! Add that to your dashboard.
With any dashboard, it’s important to get an overview at a glance. A rule of thumb with dashboards is you need to able to say in 5 seconds, if something is wrong. We try to represent failures in number and expected events in time-series metrics. Here’s what we ended up with our integration-service dashboard.
It’s still work in progress, as we’re testing between ways to display information.
The short-lived, parallel and highly scalable nature of Lambda forced us to innovate and be creative. The aforementioned approach has helped us bring clarity and visibility into our serverless systems, and I have seen it having a similar effect for other teams. Both of the tools have a free tier version, so you can easily try them out.
PS. If you have alternative ideas or would like to share your work in the monitoring field, please let me know in the comments.
Create your free account to unlock your custom reading experience.