How to sample AppSync resolver logs

AppSync has built-in logging integration with CloudWatch Logs (see for more details on the logging options). here You can set the to , or . Field resolver log level NONE ERROR ALL When the field resolver log level is set to or , you don’t get a lot of value from the AppSync logs. For example, this is what I see for a GraphQL request: NONE ERROR You can see the but have no way of figuring out what contributed towards that latency. Really, there’s not a lot you can do with the information here. latency But if you set the field resolver log level to , then you suddenly see EVERYTHING! ALL The request and response mapping for each resolver, and how long each of the fields took to resolve. This is great! But, given the amount of information that is sent to CloudWatch Logs, this has a non-trivial cost implication. As you can see below, CloudWatch Logs charges $0.50 per GB of data ingested. This is equivalent to the cost of 125,000 Query/Mutation requests to AppSync, and I bet those operations would generate a lot more than 1 GB of logs! In practice, you wouldn’t want to leave the at in a production environment. The cost would be too significant. Field resolver log level ALL Unfortunately, there is no built-in sampling support, unlike with X-Ray. Sampling resolver logs via cron jobs One workaround is to use Lambda functions to toggle the between and via a pair of cron jobs. Field resolver log level ALL ERROR This is exactly what I did on a client project recently to keep a healthy balance between cost and having visibility into what my application is doing. To do this, I configured two cron jobs (with Lambda). turnOnResolverLogging: handler: functions/toggle-resolver-log-level.handler events: - schedule: cron(8 * * * ? *) environment: APPSYNC_API_ID: !GetAtt AppsyncapiGraphQlApi.ApiId FIELD_LOG_LEVEL: ALL iamRoleStatements: - Effect: Allow Action: - appsync: GetGraphqlApi - appsync: UpdateGraphqlApi Resource: !Ref AppsyncapiGraphQlApi - Effect: Allow Action: iam:PassRole Resource: !GetAtt AppSyncLoggingServiceRole.Arn turnOffResolverLogging: handler: functions/toggle-resolver-log-level.handler events: - schedule: cron(10 * * * ? *) environment: APPSYNC_API_ID: !GetAtt AppsyncapiGraphQlApi.ApiId FIELD_LOG_LEVEL: ERROR iamRoleStatements: - Effect: Allow Action: - appsync: GetGraphqlApi - appsync: UpdateGraphqlApi Resource: !Ref AppsyncapiGraphQlApi - Effect: Allow Action: iam:PassRole Resource: !GetAtt AppSyncLoggingServiceRole.Arn These cron jobs would run on the 8th and 10th minute of each hour, giving me 2 minutes worth of resolver logs per hour. This gives me resolver logs for roughly 3% of Query/Mutation operations. The function looks like this: AppSync = ( ) appSync = AppSync() { APPSYNC_API_ID, FIELD_LOG_LEVEL } = process.env .exports.handler = (event) => { resp = appSync.getGraphqlApi({ : APPSYNC_API_ID }).promise() api = resp.graphqlApi api.logConfig.fieldLogLevel = FIELD_LOG_LEVEL api.arn api.uris api.tags appSync.updateGraphqlApi(api).promise() } const require 'aws-sdk/clients/appsync' const new const module async const await apiId const delete delete delete await To make sure I don’t accidentally change any of the other configurations, I fetch the current API configuration first. But since the request doesn’t accept , nor , I have to remove these or the request will fail validation. updateGraphqlApi arn uris tags But wait! What if I want to keep all the resolver logs in dev and only enable sampling in production? Well, you can use the plugin to conditionally exclude these two functions for the stage, like this: serverless-ifelse dev serverlessIfElse: - If: '"${self:custom.stage}" == "dev"' Exclude: - functions.turnOnResolverLogging - functions.turnOffResolverLogging The approach I outlined above is simple and gives me a sufficient amount of resolver logs without blowing up my CloudWatch cost. However, if you want the sampling to be spread more evenly across the hours then you might consider this alternative—run a cron job every few minutes and have the Lambda function: set the resolver log level to ALL busy wait for, say 10 seconds set the resolver log level back to ERROR That way, you will have resolver logs for Query/Mutation requests for 10 seconds every few minutes instead of 2 minutes every hour. Don’t forget to update log retention Another important thing to remember is to update the log retention policy for your log groups. Because by default, CloudWatch Logs will never expire your logs and you will pay $0.03 per GB per month for an ever-increasing pile of logs that are less useful with every passing minute. The best way to manage the log retention for all your log groups—past, present and future—is to deploy to your region and let it do the rest. this SAR application This SAR application would update all your existing log groups during deployment. And when new log groups are created in the future, their retention policy will also be updated automatically. Master AppSync and GraphQL If you want to learn GraphQL and AppSync with a hands-on tutorial then please check out my where you will build a Twitter clone using a combination of AppSync, Lambda, DynamoDB, Cognito and Vue.js. You will learn about different GraphQL modelling techniques, how to debug resolvers, CI/CD, monitoring and alerting and much more. new AppSync course Previously published at https://theburningmonk.com/2020/09/how-to-sample-appsync-resolver-logs/