The Trace Tail Sampling Pipeline

Tail sampling decides which traces to keep based on the full trace. The pipeline architecture, the storage requirements, and the trade-offs.

The flow

All spans are buffered in the collector for N seconds (typically 30-60).

After buffering, sampling rules evaluate the full trace. Keep error traces, slow traces, sampled healthy traces.

Storage

Buffering N seconds × spans per second = memory cost. Plan for it; size collectors accordingly.

Distributed buffering: one collector instance per trace. Routing is on trace ID.

Trade-offs

Better signal: most error/slow traces kept. Lower cost: most healthy traces dropped.

Latency to query: traces appear in the backend N seconds after they end.