part 1 : new challenges to observability part 2 : 1st party observability tools from AWS [this post] part 3 : 3rd party observability tools part 4: the future of Serverless observability In part 1 we talked about the challenges serverless brings to the table. In this post, let’s look at 1st party tools from AWS Out of the box we get a bunch of tools pro­vid­ed by AWS itself: Cloud­Watch for mon­i­tor­ing, alert­ing and visu­al­iza­tion Cloud­Watch Logs for logs X-Ray for dis­trib­uted trac­ing Ama­zon Elas­tic­Search for log aggre­ga­tion CloudWatch Logs When­ev­er you write to , those out­puts are cap­tured by the Lamb­da service and sent to Cloud­Watch Logs as logs. This is , as it’s pro­vid­ed by the plat­form. std­out one of the few background processing you get All the log mes­sages (tech­ni­cal­ly they’re referred to as ) for a giv­en function would appear in Cloud­Watch Logs under a sin­gle . events Log Group As part of a , you have many . Each con­tains the logs from one con­cur­rent exe­cu­tion (or ) of your func­tion, so there’s a one-to-one map­ping. Log Group Log Streams con­tain­er So that’s all well and good, but it’s . There’s cur­rent­ly . Whilst AWS has been improv­ing the ser­vice, it still pales in com­par­i­son to oth­er alter­na­tives on the mar­ket. not easy to search for log mes­sages in Cloud­Watch Logs no way to search the logs for mul­ti­ple func­tions at once It might suf­fice as you start out, but you’ll prob­a­bly find your­self in need of some­thing more soon after. For­tu­nate­ly, it’s straight­for­ward to get your logs out of Cloud­Watch Logs. You can stream them to Amazon’s host­ed Elas­tic­search ser­vice. But don’t expect it to be a like-for-like expe­ri­ence with your self-host­ed ELK stack though. wrote a detailed on some of the prob­lems they ran into when using Ama­zon Elas­tic­search at scale. Please give that a read if you’re think­ing about adopt­ing Ama­zon Elas­tic­search. Liz Ben­nett post Alter­na­tive­ly, you can stream the logs to a Lamb­da func­tion, and ship them to a log aggre­ga­tion ser­vice of your choice. I won’t go into detail here as I have writ­ten about it at length pre­vi­ous­ly, just go and read instead. this post You can stream logs from CloudWatch Logs to just about any log aggregation service, via Lambda. CloudWatch Metrics With Cloud­Watch, you get some basic met­rics out of the box. Invo­ca­tion count, error count, invo­ca­tion dura­tion, etc. All the basic teleme­try about the health of a func­tion. But Cloud­Watch is miss­ing some valu­able data points, such as: esti­mat­ed costs : Cloud­Watch only report this for func­tions with reserved con­cur­ren­cy con­cur­rent exe­cu­tions cold starts : Lamb­da reports this in Cloud­Watch Logs, at the end of every invo­ca­tion. Because Lamb­da invo­ca­tions are billed in 100ms blocks, a 102ms invo­ca­tion would be billed for 200ms. It will be a use­ful met­ric to see along­side to iden­ti­fy cost opti­miza­tions) billed dura­tion Invo­ca­tion Dura­tion : Lamb­da reports this in Cloud­Watch Logs too, but it’s not record­ed in Cloud­Watch mem­o­ry usage You get 6 basic metrics about the health of a function. There are ways to record and track these met­rics your­self, see on how to do that. Oth­er providers like IOPipe (more on them in the next post) would also report these data points out of the box. this post You can set up Alarms in Cloud­Watch against any of these met­rics, here are some good can­di­dates: throt­tled invo­ca­tions : set thresh­old based on % of your cur­rent region­al lim­it region­al con­cur­rent exe­cu­tions tail (95 or 99 per­centile) laten­cy against some accept­able thresh­old 4xx and 5xx errors on API Gate­way And you can set up basic dash­board in Cloud­Watch too, at $3 per month per dash­board (first 3 are free). X-Ray For dis­trib­uted trac­ing, you have X-Ray. To make the most of trac­ing, you should instru­ment your code to gain even bet­ter vis­i­bil­i­ty. Like Cloud­Watch Logs, . It’s a back­ground pro­cess­ing that the plat­form provides for you. col­lect­ing traces do not add addi­tion­al time to your function’s invo­ca­tion From the trac­ing data, X-Ray can also show you a ser­vice map like this one. X-Ray gives you a lot of insight into the run­time per­for­mance of a func­tion. How­ev­er, , the dis­trib­uted aspect is severe­ly under­cooked. As it stands, . its focus is nar­row­ly on one func­tion X-Ray cur­rent­ly doesn’t trace over API Gate­way, or asyn­chro­nous invo­ca­tions such as SNS or Kine­sis It’s good for hom­ing in on per­for­mance issues for a par­tic­u­lar func­tion. But it offers lit­tle to help you build intu­ition about how your sys­tem oper­ates . For that, I need to step away from what hap­pens inside one func­tion, and be able to look at the . as a whole entire call chain After all, when the engi­neers at Twit­ter were talk­ing about the need for , it wasn’t so much to help them debug per­for­mance issues of any sin­gle end­point, but to help them make sense of the behav­iour and of their sys­tem. A sys­tem that is essen­tial­ly one big, com­plex and high­ly con­nect­ed graph of ser­vices. observ­abil­i­ty performance With Lamb­da, this graph is going to become a lot more com­plex, more sparse and more con­nect­ed because: instead of one ser­vice with 5 end­points, you now have 5 func­tions func­tions are con­nect­ed through a greater vari­ety of mediums — SNS, Kine­sis, API Gate­way, IoT, you name it event-dri­ven archi­tec­ture has become the norm Our trac­ing tools need to help us make sense of this graph. They need to help us between our func­tions. And they need to help us fol­low data as it enters our sys­tem as a user request, and reach­es out to far cor­ners of this graph through both syn­chro­nous and asyn­chro­nous events. visu­al­ize the con­nec­tions And of course, X-Ray do not span over non-AWS ser­vices such as Auth0, or Google Big­Query, or Azure func­tions. But those of us deep in the server­less mind­set see the world through SaaS-tint­ed glass­es. We want to use the ser­vice that best address­es our needs, and them togeth­er with Lamb­da. glue At Yubl, we . Auth0, Google Big­Query, GrapheneDB, Mon­go­Lab, and Twillio to name a few. And it was great, we don’t have to be bound by what AWS offers. used a num­ber of non-AWS ser­vices from Lamb­da My good friend Raj also did a good talk at NDC on how he uses ser­vices from both AWS and Azure in his wine start­up. You can watch his talk . here And final­ly, I think of our sys­tem like a brain. Like a brain, our sys­tem is made up of: (func­tions) neu­rons (con­nec­tions between func­tions) synaps­es and (data) that flow through them elec­tri­cal sig­nals Like a brain, our sys­tem is alive, it’s con­stant­ly chang­ing and evolv­ing and it’s con­stant­ly work­ing! And yet, when I look at my dash­boards and my X-Ray traces, that’s not what I see. Instead, I see a tab­u­lat­ed list that does not reflect the and . It doesn’t help me build up any intu­itive under­stand­ing of what’s going on in my sys­tem. move­ment of data areas of activ­i­ty A brain sur­geon wouldn’t accept this as the pri­ma­ry source of infor­ma­tion. How can they pos­si­bly use it to build a men­tal pic­ture of the brain they need to cut open and oper­ate on? I should add that this is not a crit­i­cism of X-Ray, it is built the same way most observ­abil­i­ty tools are built. But maybe our tools need to evolve beyond human com­put­er inter­faces (HCI) that wouldn’t look out of place on a clip­board (the phys­i­cal kind, if you’re old enough to have seen one!). And it actu­al­ly reminds me of one of Bret Victor’s sem­i­nal talks, . stop draw­ing dead fish Net­flix made great strides towards this idea of a live dash­board with . Which they have also kind­ly . Vizcer­al open sourced Conclusions AWS pro­vides us with some decent tools out of the box. Whilst they each have their short­com­ings, they’re to get start­ed with. good enough As 1st par­ty tools, they also enjoy home field advan­tages over 3rd par­ty tools. For exam­ple, Lamb­da col­lects logs and traces with­out adding to your func­tion invo­ca­tion time. Since we can’t access the serv­er any­more, 3rd par­ty tools can­not per­form any back­ground pro­cess­ing. Instead they have to resort to workarounds or are forced to col­lect data syn­chro­nous­ly. How­ev­er, as our server­less appli­ca­tions become more com­plex, these tools need to either evolve with us or they will need to be replaced in our stack. Cloud­Watch Logs for instance, can­not search across mul­ti­ple func­tions. It’s often the first piece that need to be replaced once you have more than a dozen func­tions. In the next post, we will look at some 3rd par­ty tools such as , and . We will dis­cuss their val­ue-add propo­si­tion as well as their short­com­ings. IOPipe Dash­bird Thun­dra Like what you’re reading but want more help? I’m happy to offer my services as an and help you with your serverless project — architecture reviews, code reviews, building proof-of-concepts, or offer advice on leading practices and tools. independent consultant I’m based in and currently the only UK-based . I have nearly of with running production workloads in AWS at scale. I operate predominantly in the UK but I’m open to travelling for engagements that are longer than a week. To see how we might be able to work together, tell me more about the problems you are trying to solve . London, UK AWS Serverless Hero 10 years experience here I can also run an to help you get with your serverless architecture. You can find out more about the two-day workshop , which takes you from the basics of AWS Lambda all the way through to common operational patterns for log aggregation, distribution tracing and security best practices. in-house workshops production-ready here If you prefer to study at your own pace, then you can also find all the same content of the workshop as a I have produced for Manning. We will cover topics including: video course authentication authorization with API Gateway Cognito & & testing running functions locally & CI/CD log aggregation monitoring best practices distributed tracing with X-Ray tracking correlation IDs performance cost optimization & error handling config management canary deployment VPC security leading practices for Lambda, Kinesis, and API Gateway You can also get the face price with the code . Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP). 40% off ytcui

Flow

Dash

Amazon

BUNCH

Google

Netflix

Trace

YouTube

You need to sample debug logs in production

here are my top tips on technical writing after 8 years and 700 posts

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Serverless observability, what can you use out of the box?

Serverless observability, what can you use out of the box?

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Modeling Hierarchical Access With AppSync In 3 Simple Steps

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Modeling Hierarchical Access With AppSync In 3 Simple Steps

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps