Distributed Trace Reading

Reading a trace waterfall is about finding where time is actually spent, not which span is drawn widest at the top level. The api-gateway span's total duration (2004ms) is really just the sum of the time its downstream calls take, plus its own small overhead — it's a parent span containing child spans.

How to read the indentation: each level of indentation is a child of the span above it. orders-db is a child of orders-service, which is a child of api-gateway. To find the bottleneck, look for the deepest span whose duration is close to its parent's duration — that's the span doing the actual work, versus spans that are just waiting on their children.

Here, orders-db (1890ms) is almost exactly as long as its parent orders-service (1920ms), meaning the database call is essentially the entire cost of that branch. This tells you precisely where to look next: the SQL query itself (check EXPLAIN ANALYZE), not the network, not authentication, not the gateway.

Recognising parallel (fan-out) spans versus sequential spans is one of the highest-value trace-reading skills. The visual cue in most trace UIs (Jaeger, Zipkin, Tempo) is that sibling spans overlap horizontally in time rather than appearing one after another — the parent's total duration equals the longest overlapping child, not the sum of all children.

Critical path vocabulary: the critical path of a trace is the specific chain of spans that determines the total end-to-end duration. In a fan-out, only the slowest parallel branch is on the critical path — the faster siblings finish "for free" within that window. This has a direct engineering consequence: teams sometimes waste effort optimising a span that looks slow in isolation (200ms feels slow) but isn't actually contributing to overall latency, while the real bottleneck (fraud-check) goes unaddressed.

Communicating this precisely — "fraud-check is the critical path here, the other two are parallel and don't matter" — focuses the team's optimisation effort correctly instead of chasing the wrong span.

This is one of the most common patterns distributed tracing reveals that logs alone often hide: "death by a thousand small calls." Each individual span looks harmless (sub-millisecond), but the trace waterfall exposes that they add up — especially because each network round-trip carries fixed overhead (connection handling, serialization, TLS) regardless of how little data it carries.

Why this is hard to spot without tracing: in logs alone, you'd see 40+ short, successful entries with no obvious individual problem. The trace view, by contrast, visually shows dozens of thin bars stacked sequentially — an unmistakable pattern once you've learned to recognise it.

The N+1 fix, in this context: replace the loop of per-item calls with a single batched request (e.g. getPreferences(userIds: [...]) instead of calling getPreference(userId) once per user). This is directly analogous to the classic N+1 query problem in ORMs, just at the service-call level instead of the database level — the same diagnostic instinct ("why are there so many near-identical small operations?") applies.

span.kind is standard OpenTelemetry metadata describing the span's role: client (this service calling out to another), server (this service handling an incoming request), producer/consumer (async messaging), or internal (in-process work). Reading span.kind correctly tells you which side of a call a given span represents — essential when tracing across a service mesh where both the caller's "client span" and the callee's "server span" for the same logical call appear in the trace.

"upstream connect error or disconnect/reset before headers" is a common Envoy/service-mesh error message meaning the proxy attempted to connect to the downstream service but the connection failed or was reset before any HTTP response headers came back — this points at the downstream service being unreachable, crashed, or overloaded, rather than a problem with the request itself.

Practical use: when triaging a multi-service trace with several error spans, checking span.kind and following the error to its deepest occurrence (the true origin, not just where the error was first observed by an upstream caller) is the standard method for finding root cause rather than the first symptom.

Sampling exists because capturing 100% of traces in a high-traffic system is expensive (storage, processing, network overhead), so most systems only keep a fraction.

Head-based sampling — the decision to sample is made at the start of the trace (e.g. "randomly record 1% of all requests"), independent of what happens during the request. This is simple and cheap, but has exactly the blind spot described here: a rare slow or failing request has only a 1% chance of being captured, so investigating "why was request X slow" after the fact often turns up nothing, because that specific request was never sampled.

Tail-based sampling — spans are buffered temporarily (often by a collector like the OpenTelemetry Collector) and the sampling decision is deferred until the full trace is available, so rules like "always keep traces with errors" or "always keep traces over 1 second" can be applied. This guarantees the interesting traces are never missed, at the cost of more buffering infrastructure.

Explaining this trade-off clearly — "we're missing the failure because of head-based sampling, we should switch to tail-based sampling for error/latency traces" — turns a vague "tracing isn't helping" complaint into a specific, actionable infrastructure change.