Debugging in a complex microservices architecture can feel like navigating a maze in the dark. When a request fails or returns unexpected data, the trail often goes cold at the service boundary. You know what you sent, and you know what you got back, but the crucial "why" is locked away inside a black box. What if you could install a flight recorder on every service, transparently recording every interaction for perfect recall and inspection? This post is a deep dive into how we built such a system—a "Request Capture Engine"—from the ground up using standard, open-source components. We'll walk through the architecture, the code, and the powerful new debugging workflows it unlocks. Request Capture Engine The Core Mechanism: How to Intercept gRPC Calls The foundation of our capture system is a feature built directly into gRPC: interceptors. An interceptor is a middleware function that can "intercept" an incoming or outgoing RPC, allowing you to inspect and modify the request, the response, and the call's context. For a client-side unary (request-response) call, the concept is simple. Instead of the client calling the server directly, it calls the interceptor, which then invokes the actual RPC. This gives us the perfect hook to record the data. Here’s a simplified example of what a client interceptor looks like in Go: func UnaryPayloadCaptureInterceptor() grpc.UnaryClientInterceptor {
    return func(
        ctx context.Context,
        method string,
        req, reply interface{},
        cc *grpc.ClientConn,
        invoker grpc.UnaryInvoker,
        opts ...grpc.CallOption,
    ) error {
        // 1. Record the request payload before the call
        recordRequest(ctx, req)

        // 2. Invoke the actual RPC
        err := invoker(ctx, method, req, reply, cc, opts...)

        // 3. Record the response payload after the call
        recordResponse(ctx, reply, err)

        return err
    }
} func UnaryPayloadCaptureInterceptor() grpc.UnaryClientInterceptor {
    return func(
        ctx context.Context,
        method string,
        req, reply interface{},
        cc *grpc.ClientConn,
        invoker grpc.UnaryInvoker,
        opts ...grpc.CallOption,
    ) error {
        // 1. Record the request payload before the call
        recordRequest(ctx, req)

        // 2. Invoke the actual RPC
        err := invoker(ctx, method, req, reply, cc, opts...)

        // 3. Record the response payload after the call
        recordResponse(ctx, reply, err)

        return err
    }
} This same principle can be extended to server-side interceptors (to capture what a service receives) and, crucially, to streaming RPCs. Building the Capture Engine: A Step-by-Step Guide Capturing Payloads and the Role of OpenTelemetry To build a complete picture of an interaction, we need to capture the request and response, and we need a way to link them together. This is where distributed tracing comes in. Our system relies on OpenTelemetry, the industry standard for observability. When a request enters our system, OpenTelemetry assigns it a unique trace_id. As that request travels from one service to another, this trace_id is propagated in the gRPC metadata. Each hop in the journey is a span, with its own span_id. trace_id trace_id span span_id Our interceptor leverages this context to create a complete record: import (
    "go.opentelemetry.io/otel/trace"
    "google.golang.org/grpc"
)

func (interceptor *Interceptor) clientInterceptor() grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
        // Extract the span from the context
        span := trace.SpanFromContext(ctx)
        if !span.SpanContext().IsSampled() {
            // If tracing is not enabled for this request, do nothing
            return invoker(ctx, method, req, reply, cc, opts...)
        }

        // Create an interaction record, using IDs from the span context
        interaction := &rcappb.Interaction{
            Metadata: generateMetadata(span.SpanContext(), ...),
        }
        interaction.ReqPayload = capturePayload(ctx, req)

        err := invoker(ctx, method, req, reply, cc, opts...)

        interaction.RespPayload = capturePayload(ctx, reply)
        
        // Asynchronously push the interaction to storage
        interceptor.pushOrDiscard(ctx, interaction)

        return err
    }
}

// capturePayload convert message to proto used to store all the events
func capturePayload(ctx context.Context, msg interface{}) *capturepb.Payload {
    // Get proto information from msg
    var prMessage protoreflect.Message
    ...
    ...
    return &capturepb.Payload{
        Message: eventInBytes,
        SchemaName: string(prMessage.Descriptor().FullName())
    }
} import (
    "go.opentelemetry.io/otel/trace"
    "google.golang.org/grpc"
)

func (interceptor *Interceptor) clientInterceptor() grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
        // Extract the span from the context
        span := trace.SpanFromContext(ctx)
        if !span.SpanContext().IsSampled() {
            // If tracing is not enabled for this request, do nothing
            return invoker(ctx, method, req, reply, cc, opts...)
        }

        // Create an interaction record, using IDs from the span context
        interaction := &rcappb.Interaction{
            Metadata: generateMetadata(span.SpanContext(), ...),
        }
        interaction.ReqPayload = capturePayload(ctx, req)

        err := invoker(ctx, method, req, reply, cc, opts...)

        interaction.RespPayload = capturePayload(ctx, reply)
        
        // Asynchronously push the interaction to storage
        interceptor.pushOrDiscard(ctx, interaction)

        return err
    }
}

// capturePayload convert message to proto used to store all the events
func capturePayload(ctx context.Context, msg interface{}) *capturepb.Payload {
    // Get proto information from msg
    var prMessage protoreflect.Message
    ...
    ...
    return &capturepb.Payload{
        Message: eventInBytes,
        SchemaName: string(prMessage.Descriptor().FullName())
    }
} Of course, capturing every single payload in a high-traffic production environment could be overwhelming. It's critical to include a rate-limiter in the interceptor to sample a configurable percentage of traffic, preventing performance degradation. Handling Streaming RPCs Capturing streaming RPCs, like those used for LLM responses or large data transfers, is more complex. A stream can consist of hundreds of individual messages, and we need to capture all of them. We solve this by creating a wrapper around the standard grpc.ClientStream and grpc.ServerStream. This wrapper intercepts every SendMsg and RecvMsg call, records the message, and then passes it to the original stream. grpc.ClientStream grpc.ServerStream SendMsg RecvMsg The process looks like this: When the stream is closed (either by the client or the server), the WrappedStream takes all the recorded request and response messages and persists them as a single interaction. WrappedStream Storing the Payloads for Posterity With payloads captured, we need a place to store them. A generic object storage solution like Amazon S3 is a perfect fit—it's durable, scalable, and cost-effective. The key is a smart file naming strategy. By embedding metadata directly into the object path, we can query for data efficiently without needing a separate database index. Our structure looks like this: /request-capture-data/{trace_id}/{start_timestamp}_{source_service}_{target_service}_{span_id}.pb /request-capture-data/{trace_id}/{start_timestamp}_{source_service}_{target_service}_{span_id}.pb This allows us to instantly find all spans for a given trace_id just by listing files in a directory. trace_id A Note on Security and Privacy Storing raw request and response data is a superpower, but it comes with great responsibility. If your payloads contain Personally Identifiable Information (PII) or other sensitive data, you must handle it with care. Before persisting any data, a redaction step is essential. This process should identify sensitive fields—perhaps using proto annotations or field name conventions—and either remove them entirely or mask their values. This ensures that your debugging data remains useful for engineers without exposing sensitive user information. redaction step Making Sense of the Data: Querying and Concatenation Now that we have a mountain of valuable data, we need a way to access it. The Query Service and CLI We built a simple gRPC service that exposes an API for querying the stored payloads from object storage. // request-capture-service.proto
service RequestCapture {
    // Get all spans for a given trace
    rpc Traces(TracesRequest) returns (TracesResponse);

    // Get the full payloads for a trace or a specific span
    rpc Payloads(PayloadsRequest) returns (PayloadsResponse);
} // request-capture-service.proto
service RequestCapture {
    // Get all spans for a given trace
    rpc Traces(TracesRequest) returns (TracesResponse);

    // Get the full payloads for a trace or a specific span
    rpc Payloads(PayloadsRequest) returns (PayloadsResponse);
} While the service is useful, the primary interface for developers is a command-line tool. The CLI provides a fast, intuitive way to fetch exactly what you need. # Get all spans for a given trace ID
$ rcap traces --trace-id abc-123

# Get the full request/response payloads for that trace, formatted as JSON
$ rcap payloads --trace-id abc-123 --encoding json # Get all spans for a given trace ID
$ rcap traces --trace-id abc-123

# Get the full request/response payloads for that trace, formatted as JSON
$ rcap payloads --trace-id abc-123 --encoding json The Magic of Stream Concatenation Analyzing a stream of hundreds of LLM tokens is tedious. To solve this, we introduced the concept of a "concatenator." This is a pluggable component that knows how to merge a stream of messages into a single, coherent message. For example, our ALLMChatConcatenator takes a stream of chat deltas and merges them by role, producing a final, clean transcript of the conversation. This is configurable, and you can write custom concatenators for any streaming proto in your system. The CLI can then return both the raw stream and the concatenated result, giving you the best of both worlds. ALLMChatConcatenator Putting It All Together: From Debugging to Replay This system isn't just for looking at data; it enables entirely new workflows. Use Case 1: Advanced Debugging Having the exact, unadulterated payload is a game-changer. You can pipe the CLI's JSON output directly into tools like jq to instantly find a needle in a haystack. Use Case 1: Advanced Debugging jq # Find any response where the 'error_code' field was not 0
$ rcap payloads --trace-id abc-123 --encoding json | jq '.payloads[] | select(.resp.error_code != 0)' # Find any response where the 'error_code' field was not 0
$ rcap payloads --trace-id abc-123 --encoding json | jq '.payloads[] | select(.resp.error_code != 0)' Use Case 2: Building a High-Fidelity Traffic Replay System This is perhaps the most powerful use case. The captured payloads are a perfect, high-fidelity record of production traffic. You can build a system to "replay" these payloads against a staging or local instance of a service. This is the most reliable way to reproduce complex bugs that only appear with specific, hard-to-guess data patterns. You can even pipe the output of the rcap tool directly into a gRPC CLI to replay a request with a single command. Use Case 2: Building a High-Fidelity Traffic Replay System rcap Use Case 3: Powering Interactive Debugging UIs While the command line is powerful, the Request Capture Engine's API can also be the backbone for rich, interactive web UIs. Imagine an experimentation UI for your service that directly integrates with the payload store. This unlocks several advanced workflows: Use Case 3: Powering Interactive Debugging UIs Search and Load: A developer can paste a trace_id from a production error into the UI, which then calls the Request Capture Engine service to fetch the exact request payload and load it into an editor.
Modify and Replay: The developer can then tweak the request parameters in the UI and replay the request against a development or staging environment to test a fix.
A/B Comparison: The UI could load a single production request and replay it against two different versions of a service simultaneously, providing a side-by-side comparison of the responses to validate a change.
Shareable Debug Sessions: By integrating with the payload store, a developer can share a URL that links directly to a specific trace, making it easy to collaborate on debugging complex issues. Search and Load: A developer can paste a trace_id from a production error into the UI, which then calls the Request Capture Engine service to fetch the exact request payload and load it into an editor. Search and Load: trace_id Modify and Replay: The developer can then tweak the request parameters in the UI and replay the request against a development or staging environment to test a fix. Modify and Replay: A/B Comparison: The UI could load a single production request and replay it against two different versions of a service simultaneously, providing a side-by-side comparison of the responses to validate a change. A/B Comparison: Shareable Debug Sessions: By integrating with the payload store, a developer can share a URL that links directly to a specific trace, making it easy to collaborate on debugging complex issues. Shareable Debug Sessions: This turns the payload capture system from a simple debugging tool into a foundational platform for developer tooling. Conclusion By combining gRPC interceptors, OpenTelemetry, and a simple object storage layout, we've built a "Request Capture Engine" for our microservices. It has transformed debugging from a frustrating exercise in guesswork into a deterministic process of inspection. The ability to see the exact data that caused a problem, and to replay that data on demand, has dramatically accelerated our ability to build and maintain reliable systems. Building your own payload capture system is an achievable and high-impact investment that will pay dividends in developer productivity and service reliability.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Amazon

Building a Better Debugging Experience: A Deep Dive into Capturing and Replaying gRPC Traffic

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Beyond Keywords: Engineering a Production-Ready Agentic Search Framework in Go

10 Must Have Chrome Extensions for a Web Developer

The Noonification: Can Mankind Survive as an Interplanetary Species? (10/20/2022)

132 Stories To Learn About Performance

3 Golang Pitfalls Every Developer Needs to Know

🤩 24 Coolest VSCode Extensions That Will Rock Your World 🛠

Beyond Keywords: Engineering a Production-Ready Agentic Search Framework in Go

10 Must Have Chrome Extensions for a Web Developer

The Noonification: Can Mankind Survive as an Interplanetary Species? (10/20/2022)

132 Stories To Learn About Performance

3 Golang Pitfalls Every Developer Needs to Know

🤩 24 Coolest VSCode Extensions That Will Rock Your World 🛠

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps