The Burst Buffer Before Eviction

Telemetry data should buffer briefly before eviction. The pattern, the storage, and the tail-sampling enabling it.

The idea

Burst buffer before eviction is the technique of holding incoming spans briefly so tail-based sampling decisions can use the full trace. Without buffering, the collector decides per-span; with buffering, the decision is per-trace at the end of the buffer window. The technique enables much smarter sampling.

What the technique looks like:

• Hold incoming spans for 60 seconds.: Spans arrive at the collector and go into a buffer. The buffer holds them for the configured duration (60 seconds is typical) before any sampling decision.
• Decisions about retention happen at the end of the buffer window.: When the buffer window expires for a trace, the sampling decision is made. By then, all (or most) spans of the trace have arrived; the decision uses the full trace.
• Allows tail-based decisions.: The decision can use information from anywhere in the trace. "Keep this trace if any span has an error" is a valid rule; the decision sees all spans.
• Keep this trace if it had errors; drop if not.: Specific patterns are now possible. Errors get kept; healthy traces sample at a low rate; the team's storage focuses on the high-value traces.
• Other patterns: latency-based, attribute-based.: "Keep traces over 1 second"; "keep traces from premium customers". The patterns are flexible because the decision has the full trace context.

The technique is what makes tail sampling practical at scale. Without the buffer, tail sampling is theoretical.

Storage

The buffer's storage characteristics matter. Memory is fast but expensive; disk is cheap but slow. Distributed buffering enables horizontal scale.

• Memory mostly.: The buffer typically lives in collector memory. Memory is fast; the buffer's access pattern (write on receive, read at decision time) fits memory well.
• Spill to local disk if over capacity.: When memory pressure is high, spill to local disk. The disk spill is slower but prevents OOM; it is a safety valve, not the primary storage.
• Distributed buffering.: At large scale, one collector cannot buffer all traces. Multiple collectors share the load; each collector handles a partition of traces.
• Route by trace ID.: The partition strategy routes by trace ID. All spans of a single trace land on the same collector; the buffer for the trace is complete on that collector. Without this routing, traces fragment across collectors.
• So all spans of a trace land on the same collector.: The locality is essential. The sampling decision needs the full trace; spans on different collectors cannot make joint decisions easily.

The storage architecture supports the buffering at scale. Without the right architecture, the buffer becomes a bottleneck.

Trade-offs

The buffer has costs. Memory consumption, latency in trace appearance, and operational complexity all are real. The trade-off matters.

• Cost: collector memory.: The collector needs memory for the buffer. The size scales with buffer duration times span rate. High-volume systems need significant collector memory.
• Sized to the buffer duration times span rate.: A 60-second buffer with 1000 spans per second per collector requires 60,000 spans in memory. Each span has metadata; the total memory is proportional. The team plans the collector sizing.
• Latency: spans appear in the backend N seconds after their trace ends.: The buffer adds latency. A span produced at time T appears in the backend at time T plus buffer duration. For 60-second buffers, the latency is 60 seconds.
• Acceptable for most uses.: Trace investigation typically does not need real-time spans. The 60-second latency is acceptable for postmortem investigation and dashboards. Real-time alerting may need different paths.
• Operational complexity.: Buffering, routing, decision rules all add complexity. The collector configuration is more involved; debugging is harder when something goes wrong. The trade-off is real but bounded.

Burst buffer before eviction is one of those tracing infrastructure patterns that enables sophisticated sampling at scale. Nova AI Ops integrates with tracing collectors, surfaces buffer health and decision patterns, and helps teams configure the buffering that matches their workload.