A Guide to Local Debugging with Datadog Agentby@hacker5281989
5,703 reads
5,703 reads

A Guide to Local Debugging with Datadog Agent

by Dhivya DandapaniMarch 21st, 2023
Read on Terminal Reader
Read this story w/o Javascript

Too Long; Didn't Read

The Datadog tool is used to visualize production logs. It can be installed on a MacOSX machine. The tool is written in Java and can be used to monitor Java applications. It is free and available for free on the company's website. The instructions below are for the local installation.
featured image - A Guide to Local Debugging with Datadog Agent
Dhivya Dandapani HackerNoon profile picture

Let’s start with “why” should one even bother with setting up Datadog on the local machine. I can share my motivation to set it up

My strong ‘why’:trying to make sense of the flaky tests:

We had an extensive end-to-end test suite with high coverage. But sadly, the tests grew flakier and flakier over time. To my dismay, when the tests were run in isolation, they’d pass. When we ran the entire test suite, many tests would start failing. The application logs corresponding to the tests should help us debug. But I couldn’t even understand where the logs for a particular test started and ended, let alone make sense of them.

Datadog to the rescue 🦸!

We have been using Datadog in our project to visualize the production logs and we found it super useful. That’s when I had an epiphany! What if I set up Datadog locally to dissect these logs? And that’s what triggered the contents of this blog post.

Our application is written in Java and my local machine runs on MacOS. You could tweak the below steps to suit your need.

What do you need to set up:

components needed for set-up

  1. Sign up for Datadog here. Our scope here is only local debugging, so a free trial should suffice.
  2. Datadog Agent is the software that monitors the host on which it runs. Settings like service name, environment, etc. can be made in the agent.
  3. dd-trace-java, Datadog's APM (App Performance Monitoring) client Java library. dd-trace-java is a jar that contains APIs to automatically or manually trace and profile Java applications.

Step 1 — Sign up process:

  • After giving your email id and other details, you would be prompted to choose your tech stack on the next page.

  • after you click ‘Next’, comes the most important part. You would be given the ‘agent setup’ instructions. My local machine is MacOSX, so I chose it appropriately. Please note down the DD_API_KEY and DD_SITE values, as they will be used in the subsequent steps.

instructions for installing the agent

Step 2— Datadog Agent Installation:

For MacOS, you could install via brew (package manager for MacOS). Use the commands given here for installation. ¹

open the downloaded brew pkg & install using the wizard

  • Customize the datadog-agent as shown below


  • After the above step is completed, you’ll be asked to ‘Finish’ in the Datadog-web-UI.


  • You’ll be prompted to add any integrations. You could add JavaDocker etc (as per your project needs).

Step 3 — Install Datadog tracer:

  • Download the dd-java-agent jar from here ²

  • For your Java application, make sure you’re using a version of Java that comes with an in-built FlightRecorder. This is needed for tracing. OpenJdK8 comes with an inbuilt FlightRecorder. If you’re using Oracle Java, you need a commercial license to use **FlightRecorder.**³

Running your application:

  • add the VM options for our java application:
-javaagent:"/Users/home/Downloads/dd-java-agent.jar" -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.agent.port=8126 -Ddd.service=ecommerce-service -Ddd.env=local -XX:FlightRecorderOptions=stackdepth=256

JVM args for java application

  • If you are using a gradle task, you can include the jvmArgs as below:

JVM args for Gradle task

jvmArgs = ["-javaagent:/Users/home/Downloads/dd-java-agent.jar", "-XX:FlightRecorderOptions=stackdepth=256", "-Ddd.profiling.enabled=true", "-Ddd.logs.injection=true", "-Ddd.agent.port=8126", "-Ddd.service=ecommerce-service", "-Ddd.env=local"]

  • Now start your Java application. If the tracing is happening correctly, you should see the below logs:

Datadog tracing started — logs

Logging in Datadog:

  • To enable logs to be streamed to Datadog, follow the instructions in the Datadog Web UI. ⁴

configure log source

  • For macOS— Config directory is ~/.datadog-agent/conf.d/java.d and the file is present at ~/.datadog-agent/conf.d/java.d/conf.yaml

an example ‘conf.yaml’ file from local

  • and now, the logsshould start flowing in Datadog.

Datadog logs

Some Tips:

  • Add appropriate debuglogs at the beginning and end of each method in your application. It’s very useful while debugging.

  • Add a mechanism to inject span idand trace id. It’s an amazing lifesaver and puts the logs in proper context.⁵


Once I had the Datadog agent up and running locally, I was able to view all the requests contextually.

My tests — before debugging & after using Datadog

And voila! All of my end-to-end tests are passing now 🙂 Happy logging!

Additional Info:

  • ¹Alternatively, you could also download the dmg package and install it. This link can also be obtained in the above ‘Agent Setup’ step (refer to screenshot)

  • ²Direct download the dd-trace-java jar.

  • ³ Starting from JDK 11, we may use FlightRecorderwithout activating anything. More details can be found here.

  • ⁴More details are here.

  • ⁵If your application is Java, you could you this library spring-cloud-sleuth for adding span_id and trace_id to the request.