With the advent of Cloudflare’s new Enterprise Log Share (ELS) REST API, it is now easy to download and inspect a zone’s access logs by a single RayID (Cloudflare’s fancy term for a request ID), a defined time period, or from the last RayID downloaded. In our case, we ship all of our other logs to Sumo Logic so everyone can analyze and parse them in any way imaginable!
Unfortunately, there seems to be no cut-and-dry way to integrate the two. Thankfully, AWS Lambda functions are now able to be run on a scheduled interval using CloudWatch Scheduled Events — sort of like you would a cron job on a server.
There are a couple of clear benefits of using a Lambda function instead of a job on a physical/virtual server you run yourself:
First, you’re going to want to create a new HTTP source to ingest the Cloudflare logs. Set the source category, host, etc. at SumoLogic to make the configuration of the Lambda function easier.
Next, create a new Serverless function:
$ mkdir -p ~/cloudflareToSumoLogic
$ cd ~/cloudflareToSumoLogic
$ serverless create \--name cloudflareToSumoLogic \--template aws-nodejs \--path ~/cloudflareToSumoLogic
Let’s walk through the serverless.yml
configuration file:
functions:
# This is the name of the Lambda function cloudflareToSumoLogic:
**_\# This is the name of the handler; this is the "main"_ \# _function called by Lambda_**
handler: index.handler
**_\# This function was written to be run every one minute
# Any adjustments will require code changes..._**
events:
- schedule: rate(1 minute)
**_\# Required environment variables:_**
environment:
**\# Grab this URL from the Sumo Logic source you just created**
SUMO\_ENDPOINT: ...
**\# Get these values from the Cloudflare admin UI**
CLOUDFLARE\_ZONE\_ID: ...
CLOUDFLARE\_AUTH\_EMAIL: ...
CLOUDFLARE\_AUTH\_KEY: ...
Finally, you’re going to need the function:
Now, try it out!
$ serverless invoke local -f cloudflareToSumoLogic
startTime: 2017-02-11T15:07:00.000ZendTime: 2017-02-11T15:08:00.000Z
cloudflareOpts: { method: 'GET',hostname: 'api.cloudflare.com',path: '/client/v4/zones/.../logs/requests?start=1486825620&end=1486825680',headers:{ 'X-Auth-Email': '...','X-Auth-Key': '...' } }
res.statusCode: 200
res.headers: { date: 'Sat, 11 Feb 2017 15:38:40 GMT','content-type': 'application/json','transfer-encoding': 'chunked',connection: 'close','set-cookie': [ '__cfduid=...; expires=Sun, 11-Feb-18 15:38:39 GMT; path=/; domain=.cloudflare.com; HttpOnly' ],vary: 'Accept-Encoding','x-cf-request-id': '00000000-0000-0000-0000-000000000000','strict-transport-security': 'max-age=31536000','served-in-seconds': '0.355',server: 'cloudflare-nginx','cf-ray': '...-ORD' }
Log events: 659
Everything look good? It’s time to deploy:
$ serverless deploy
Serverless: Packaging service...Serverless: Uploading CloudFormation file to S3...Serverless: Uploading service .zip file to S3 (9.45 KB)...Serverless: Updating Stack...Serverless: Checking Stack update progress........................Serverless: Stack update finished...Serverless: Removing old service versions...
Service Informationservice: cloudflareToSumoLogicregion: us-east-1api keys:Noneendpoints:Nonefunctions:cloudflareToSumoLogic
Now that your logs are being ingested every minute, you can run the following query and you should be able to start seeing results like the screenshot above:
_sourceCategory=...| json auto| toLong(num(timestamp)) as timestamp| timestamp as _messageTime| fields - timestamp| fields - _raw
Replace _sourceCategory
with the HTTP source category that you created earlier for your Cloudflare logs. This snippet parses the JSON lines automatically, translates the log timestamp timestamp
to a time object, replaces the _messageTime
with the log timestamp, then remove the raw JSON object and its ingest timestamp.