It’s common practice to set log level to WARNING for production due to traffic volume. This is because we have to consider various cost factors:
- cost of logging : CloudWatch Logs charges $0.50 per GB ingested. In my experience, this is often much higher than the Lambda invocation costs
- cost of storage : CloudWatch Logs charges $0.03 per GB per month, and its default retention policy is Never Expire! A common practice is to ship your logs to another log aggregation service and to set the retention policy to X days. See this post for more details.
- cost of processing : if you’re processing the logs with Lambda, then you also have to factor in the cost of Lambda invocations.
But, doing so leaves us without ANY debug logs in production. When a problem happens in production, you won’t have the debug logs to help identify the root cause.
Instead you have to waste precious time to deploy a new version of your code to enable debug logging. Not to mention that you shouldn’t forget to disable debug logging when you deploy the fix.
With microservices, you often have to do this for more than one service to get all the debug messages you need.
All these, increases the mean time to recovery (MTTR) during an incident. That’s not what we want.
It doesn’t have to be like that.
There is a happy middle ground between having no debug logs and having all the debug logs. Instead, we should sample debug logs from a small percentage of invocations.
I demoed how to do this in the Logging chapter of my video course Production-Ready Serverless. You need two basic things:
- a logger that lets you to change the logging level dynamically, e.g. via environment variables.
- a middleware engine such as middy
With Lambda, I don’t need most of the features from a fully-fledged logger such as pino. Instead, I prefer to use a simple logger module like this one. It’s written in a handful of lines and gives me the basics:
- structured logging with JSON
- ability to log at different levels
- ability to control the log level dynamically via environment variables
Using middy, I can create a middleware to dynamically update the log level to DEBUG. It does this for a configurable percentage of invocations. At the end of the invocation the middleware would restore the previous log level.
You might notice that we also have some special handling for when the invocation errs.
This is to ensure we capture the error with as much context as possible, including:
- the unique AWS Request ID
- the invocation event, so we can replay the invocation event locally and debug the problem
- the error message and stacktrace
Having debug logs for a small percentage of invocation is great. But when you’re dealing with microservices you need to make sure that your debug logs cover an entire call chain.
That is the only way to put together a complete picture of everything that happened on that call chain. Otherwise, you will end up with fragments of debug logs from many call chains but never the complete picture of one.
You can do this by forwarding the decision to turn on debug logging as a correlation ID. The next function in the chain would respect this decision, and pass it on. See this post for more detail.
So that’s it, another pro tip on how to build observability into your serverless application. If you want to learn more about how to go all in with serverless, check out my 10-step guide here.
Until next time!
Like what you’re reading but want more help? I’m happy to offer my services as an independent consultant and help you with your serverless project — architecture reviews, code reviews, building proof-of-concepts, or offer advice on leading practices and tools.
I’m based in London, UK and currently the only UK-based AWS Serverless Hero. I have nearly 10 years of experience with running production workloads in AWS at scale. I operate predominantly in the UK but I’m open to travelling for engagements that are longer than a week. To see how we might be able to work together, tell me more about the problems you are trying to solve here.
I can also run an in-house workshops to help you get production-ready with your serverless architecture. You can find out more about the two-day workshop here, which takes you from the basics of AWS Lambda all the way through to common operational patterns for log aggregation, distribution tracing and security best practices.
If you prefer to study at your own pace, then you can also find all the same content of the workshop as a video course I have produced for Manning. We will cover topics including:
- authentication & authorization with API Gateway & Cognito
- testing & running functions locally
- log aggregation
- monitoring best practices
- distributed tracing with X-Ray
- tracking correlation IDs
- performance & cost optimization
- error handling
- config management
- canary deployment
- leading practices for Lambda, Kinesis, and API Gateway
You can also get 40% off the face price with the code ytcui. Hurry though, this discount is only available while we’re in Manning’s Early Access Program (MEAP).