It’s com­mon prac­tice to set log lev­el to WARNING for pro­duc­tion due to traf­fic vol­ume. This is because we have to con­sid­er var­i­ous cost fac­tors: : charges $0.50 per GB ingest­ed. In my expe­ri­ence, this is often much high­er than the Lamb­da invo­ca­tion costs cost of log­ging Cloud­Watch Logs : charges $0.03 per GB per month, and its default reten­tion pol­i­cy is ! A com­mon prac­tice is to ship your logs to anoth­er log aggre­ga­tion ser­vice and to set the reten­tion pol­i­cy to X days. See for more details. cost of stor­age Cloud­Watch Logs Nev­er Expire this post : if you’re pro­cess­ing the logs with , then you also have to fac­tor in the cost of invo­ca­tions. cost of pro­cess­ing Lamb­da Lamb­da But, doing so leaves us with­out ANY debug logs in pro­duc­tion. When a prob­lem hap­pens in pro­duc­tion, you won’t have the debug logs to help iden­ti­fy the root cause. Instead you have to waste pre­cious time to deploy a new ver­sion of your code to enable debug log­ging. Not to men­tion that you shouldn’t for­get to dis­able debug log­ging when you deploy the fix. With microser­vices, you often have to do this for more than one ser­vice to get all the debug mes­sages you need. All these, . That’s not what we want. increas­es the mean time to recov­ery (MTTR) dur­ing an inci­dent It doesn’t have to be like that. There is a hap­py mid­dle ground between hav­ing no debug logs and hav­ing all the debug logs. Instead, we should sam­ple debug logs from a small per­cent­age of invo­ca­tions. I demoed how to do this in the chap­ter of my video course . You need two basic things: Log­ging Pro­duc­tion-Ready Server­less a log­ger that lets you to change the log­ging lev­el dynam­i­cal­ly, e.g. via envi­ron­ment vari­ables. a mid­dle­ware engine such as mid­dy With , I don’t need most of the fea­tures from a ful­ly-fledged log­ger such as . Instead, I pre­fer to use a sim­ple log­ger mod­ule like . It’s writ­ten in a hand­ful of lines and gives me the basics: Lamb­da pino this one struc­tured log­ging with JSON abil­i­ty to log at dif­fer­ent lev­els abil­i­ty to con­trol the log lev­el dynam­i­cal­ly via envi­ron­ment vari­ables Using , I can cre­ate a to dynam­i­cal­ly update the log lev­el to DEBUG. It does this for a con­fig­urable per­cent­age of invo­ca­tions. At the end of the invo­ca­tion the mid­dle­ware would restore the pre­vi­ous log lev­el. mid­dy mid­dle­ware You might notice that we also have some spe­cial han­dling for when the invo­ca­tion errs. This is to ensure we cap­ture the error with as much con­text as pos­si­ble, includ­ing: the unique AWS Request ID the invo­ca­tion event, so we can and debug the prob­lem replay the invo­ca­tion event local­ly the error mes­sage and stack­trace Hav­ing debug logs for a small per­cent­age of invo­ca­tion is great. But when you’re deal­ing with microser­vices you need to make sure that your debug logs cov­er an entire call chain. That is the only way to put togeth­er a com­plete pic­ture of every­thing that hap­pened on that call chain. Oth­er­wise, you will end up with frag­ments of debug logs from many call chains but nev­er the com­plete pic­ture of one. You can do this by for­ward­ing the deci­sion to turn on debug log­ging as a cor­re­la­tion ID. The next func­tion in the chain would respect this deci­sion, and pass it on. See for more detail. this post So that’s it, anoth­er pro tip on how to build observ­abil­i­ty into your server­less appli­ca­tion. If you want to learn more about how to go all in with server­less, check out my 10-step guide . here Until next time! Like what you’re reading but want more help? I’m happy to offer my services as an and help you with your serverless project — architecture reviews, code reviews, building proof-of-concepts, or offer advice on leading practices and tools. independent consultant I’m based in and currently the only UK-based . I have nearly of with running production workloads in AWS at scale. I operate predominantly in the UK but I’m open to travelling for engagements that are longer than a week. To see how we might be able to work together, tell me more about the problems you are trying to solve . London, UK AWS Serverless Hero 10 years experience here I can also run an to help you get with your serverless architecture. You can find out more about the two-day workshop , which takes you from the basics of AWS Lambda all the way through to common operational patterns for log aggregation, distribution tracing and security best practices. in-house workshops production-ready here If you prefer to study at your own pace, then you can also find all the same content of the workshop as a I have produced for Manning. We will cover topics including: video course authentication authorization with API Gateway Cognito & & testing running functions locally & CI/CD log aggregation monitoring best practices distributed tracing with X-Ray tracking correlation IDs performance cost optimization & error handling config management canary deployment VPC security leading practices for Lambda, Kinesis, and API Gateway You can also get the face price with the code . Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP). 40% off ytcui

Chain

Amazon

Trace

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

ML Practitioners - Ready to Level Up your Skills?

You need to sample debug logs in production

Too Long; Didn't Read

Companies Mentioned

Coin Mentioned

@theburningmonk

RELATED STORIES