The request we could not follow across four services

A customer sent us a screenshot of a 500 error with a timestamp. That was all we had. The request had passed through an API gateway, an auth service, an orders service, and a payments service, and each of them logged beautifully in its own format, into its own index, with no shared identifier anywhere. Finding that one request meant grepping four log stores by timestamp window and guessing which lines belonged together.

We got it eventually, about ninety minutes later, by correlating on a user ID that happened to appear in three of the four services. The fourth logged only an internal account key, so we joined through a database query. Ninety minutes to answer "what happened to this one request." That's not an observability problem you fix with more logs. We had plenty of logs. We had no way to connect them.

The missing piece was a correlation ID: one identifier generated at the edge, passed through every hop, and written into every log line. It sounds almost too simple to matter until you've had it, and then you can't work without it. One field, and the ninety-minute archaeology becomes a single query returning the full path of a request in order.

Retrofitting it was less about technology than about boundaries. Every internal HTTP call had to propagate the header. Every async message needed the ID carried in its envelope, which is where most of the work was, because queues are exactly where causal chains normally get severed. Every log statement needed the ID in structured context rather than interpolated into a message string, so it could be indexed instead of regexed. We used the W3C trace context header so the same ID also fed our tracing backend for free.

What surprised me was how much the shape of the system changed once we could see it. Requests we thought made two downstream calls made nine. A retry loop we didn't know about doubled traffic to one service. A cache we assumed was hot was being missed constantly, visible instantly in the trace as a gap.

Logs describe events. Traces describe causality. If you can't follow one request end to end, you're not debugging a distributed system, you're interviewing witnesses who never met.

– Sergey Shinder