As a backend developer working with microservices for the past few years, one truth has become painfully obvious: debugging production issues across distributed systems can feel like detective work in the dark. You’ve got services calling services, sometimes dozens deep. A user clicks a button on the UI, and 15 microservices spin into action. If something breaks — or worse, just slows down — figuring out where and why can chew up hours. This is exactly why observability matters. And if you’re building or maintaining microservices in 2024, OpenTelemetry is the tool you want in your corner. What Even Is Observability, Really? Observability is more than just logs. It’s about understanding why your system is behaving a certain way, not just what it’s doing. At the core, we’re talking about three pillars: Logs – Raw events, helpful for debugging.Metrics – Numbers you can track over time (e.g. request count, CPU).Traces – End-to-end request flows across services (aka your distributed "call stack"). Logs – Raw events, helpful for debugging. Metrics – Numbers you can track over time (e.g. request count, CPU). Traces – End-to-end request flows across services (aka your distributed "call stack"). Traditional monitoring tools mostly focus on metrics and logs, but tracing is the real game-changer for microservices. Why We Picked OpenTelemetry We experimented with several observability stacks — Datadog, New Relic, Prometheus, Jaeger, Zipkin — but they all had one problem: either they were vendor-locked or lacked consistency across languages. OpenTelemetry (OTel) checked all our boxes: Open-source, under CNCFWorks across languages (we use Node.js, Go, and Python)Vendor-neutral — export to Grafana, Jaeger, New Relic, etc.Supported by everyone in the industry (literally: AWS, GCP, Microsoft, etc.) Open-source, under CNCF Works across languages (we use Node.js, Go, and Python) Vendor-neutral — export to Grafana, Jaeger, New Relic, etc. Supported by everyone in the industry (literally: AWS, GCP, Microsoft, etc.) How We Use OpenTelemetry in Node.js Microservices Let me walk you through how we actually instrumented a real service. Let’s say we’ve got a simple User Service built in Node.js using Express. It exposes an endpoint `/users` that fetches user data. Below are the steps. Step 1: Install Dependencies Step 1: Install Dependencies npm install @opentelemetry/api \
@opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http npm install @opentelemetry/api \
@opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http We’re going to export traces via OTLP to a local Jaeger instance. Step 2: Create tracing.js to Initialize OpenTelemetry Step 2: Create tracing.js to Initialize OpenTelemetry JavaScript: tracing.js JavaScript: tracing.js const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { Resource } = require('@opentelemetry/resources');
const traceExporter = new OTLPTraceExporter({
url: 'http://localhost:4318/v1/traces',
});

const sdk = new NodeSDK({
traceExporter,
instrumentations: [getNodeAutoInstrumentations()],
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'user-service',})
});
sdk.start(); const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { Resource } = require('@opentelemetry/resources');
const traceExporter = new OTLPTraceExporter({
url: 'http://localhost:4318/v1/traces',
});

const sdk = new NodeSDK({
traceExporter,
instrumentations: [getNodeAutoInstrumentations()],
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'user-service',})
});
sdk.start(); Step 3: Add It to Your Entry File Step 3: Add It to Your Entry File JavaScript: Index.js

require('./tracing'); // Always load this first

const express = require('express');
const app = express();

app.get('/users', (req, res) => {
 res.json([{ id: 1, name: "Alice" }, { id: 2, name: "Bob" }]);
});

app.listen(3000, () => console.log("User service running on port 3000")); JavaScript: Index.js

require('./tracing'); // Always load this first

const express = require('express');
const app = express();

app.get('/users', (req, res) => {
 res.json([{ id: 1, name: "Alice" }, { id: 2, name: "Bob" }]);
});

app.listen(3000, () => console.log("User service running on port 3000")); Our service is now exporting traces. Step 4: Spin Up Jaeger Locally (or Use Grafana Tempo) Step 4: Spin Up Jaeger Locally (or Use Grafana Tempo) Here’s how we test locally: docker run -d --name jaeger \
 -e COLLECTOR_OTLP_ENABLED=true \
 -p 4318:4318 -p 16686:16686 \
 jaegertracing/all-in-one:latest docker run -d --name jaeger \
 -e COLLECTOR_OTLP_ENABLED=true \
 -p 4318:4318 -p 16686:16686 \
 jaegertracing/all-in-one:latest Visit http://localhost:16686 to view your traces. Visit http://localhost:16686 to view your traces. http://localhost:16686 Chaining Traces Across Services Now say you have another service — order-service — that calls user-service. If both are instrumented with OpenTelemetry, you’ll get a full trace of the user request hopping between them. And the best part? OpenTelemetry handles trace context propagation via HTTP headers automatically. You don’t have to manually pass trace IDs between services. Adding Custom Spans for Business Logic Sometimes auto-instrumentation isn’t enough. For example, if you want to trace a DB query or external API call: const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('user-service');

app.get('/users', async (req, res) => {
 const span = tracer.startSpan('fetch-user-data');
 try {
  const users = await fetchUsersFromDB();
  res.json(users);
 } catch (err) {
  span.recordException(err);
  throw err;
 } finally {
  span.end();
 }
}); const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('user-service');

app.get('/users', async (req, res) => {
 const span = tracer.startSpan('fetch-user-data');
 try {
  const users = await fetchUsersFromDB();
  res.json(users);
 } catch (err) {
  span.recordException(err);
  throw err;
 } finally {
  span.end();
 }
}); This is super helpful when you want to track performance of specific business logic. Best Practices We've Learned the Hard Way Best Practices We've Learned the Hard Way 1. Use Semantic Conventions Use Semantic Conventions Instead of inventing your own attribute names, stick with the OpenTelemetry semantic conventions. These make your traces easier to understand and compatible with tools like Grafana, Tempo, etc. Example: JavaScript
span.setAttribute("http.method", req.method);
span.setAttribute("http.route", req.path); JavaScript
span.setAttribute("http.method", req.method);
span.setAttribute("http.route", req.path); 2. Sample Wisely 2. Sample Wisely If you trace every single request, your system will drown in data. Use trace sampling (e.g. 10%, or only errors). JavaScript const sdk = new NodeSDK ({
 sampler: new TraceIdRatioBasedSampler (0.1), // 10% sampling
}); const sdk = new NodeSDK ({
 sampler: new TraceIdRatioBasedSampler (0.1), // 10% sampling
}); 3. Use OpenTelemetry Collector in Production 3. Use OpenTelemetry Collector in Production Don’t export telemetry data directly from your services to your backend. Route it through the OpenTelemetry Collector — it gives you buffering, batching, retries, and format conversion. 4. Don't Log PII in Spans 4. Don't Log PII in Spans This one’s critical. Be super careful not to store user names, emails, credit card info, etc. in span attributes or logs. Stick to metadata and identifiers. Where This Has Helped Us Most Debugging latency issues: Seeing full traces across 4–5 microservices helped us identify bottlenecks in minutes.Identifying retry storms: We spotted a service calling another in a loop with retries, something we wouldn’t have caught via logs.Deployment regressions: Comparing traces from one version to the next showed us exactly what changed. Debugging latency issues: Seeing full traces across 4–5 microservices helped us identify bottlenecks in minutes. Identifying retry storms: We spotted a service calling another in a loop with retries, something we wouldn’t have caught via logs. Deployment regressions: Comparing traces from one version to the next showed us exactly what changed. Bonus: Tracing in a Multi-Language Stack We're using Node.js for some services, Go for others. OpenTelemetry made it easy to instrument both and send all data to a single place — Jaeger for dev, Grafana Cloud in staging/prod. No vendor lock-in. No mismatch in trace formats. Just pure visibility. Conclusion: If You're Building Microservices, Start with Observability Conclusion: If You're Building Microservices, Start with Observability Microservices give us scale and flexibility, but they also bring complexity. Without proper observability, you're flying blind.OpenTelemetry has become a core part of our architecture, not just for debugging, but for optimizing performance, reliability, and ultimately — the user experience. Microservices give us scale and flexibility, but they also bring complexity. Without proper observability, you're flying blind. OpenTelemetry has become a core part of our architecture, not just for debugging, but for optimizing performance, reliability, and ultimately — the user experience. If anyone not using it yet, I strongly recommend giving it a shot. Even a basic setup with Jaeger and a couple services will make you wonder how you ever lived without it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Datadog

Beyond the Perimeter: Architecting Trust in the Cloud-Native Era by Mohit Kumar Singh

Too Long; Didn't Read

Make resilience your competitive advantage

Microservices Observability: Leveraging OpenTelemetry in Real-World Systems by Gajinder Sharma

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Beyond the Perimeter: Architecting Trust in the Cloud-Native Era by Mohit Kumar Singh

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

Beyond the Perimeter: Architecting Trust in the Cloud-Native Era by Mohit Kumar Singh

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps