The Trace Attribute Cost Model
Each attribute on a span costs storage. The model, the budget, and the high-value attributes that pay their keep.
The cost model
Trace attribute cost model is the discipline of understanding how much each attribute costs across the team's trace volume. Attributes feel free at the application layer; in aggregate they are not. Understanding the cost model produces deliberate attribute choices.
What the cost model captures:
- Each attribute key and value adds bytes to every span.: Adding an attribute to a span increases the bytes per span. The increase is small per span; in aggregate it is significant.
- Multiply by spans per second.: Each span has the attribute; the cost is bytes-per-attribute times spans-per-second. The total bytes flowing through the trace pipeline scales with both factors.
- That is the cost.: The cost is real. Storage cost, ingestion cost, network cost all scale with bytes. The team's trace bill includes the attribute overhead.
- 1 KB attribute times 1000 spans per second times 30 days equals 2.5 TB.: The math illustrates the magnitude. A 1 KB attribute (large but not extreme) at 1000 spans per second produces 2.5 TB of additional storage per month. The cost is real money.
- Real money.: Vendor pricing typically charges per GB ingested or stored. The attribute math translates directly to dollars; teams can quantify the cost of each attribute.
The cost model is the foundation. Without it, attribute decisions are made without economic awareness.
The budget
Each service should have an attribute budget. Above the budget, the team decides what to drop. The budget enforces deliberate choices; the discipline produces sustainable cost.
- 10 to 15 attributes per span.: The typical budget is 10 to 15 attributes per span. Below this, attributes feel free; above it, the cost is meaningful.
- Above that, decide what to drop.: The team's budget review identifies attributes that are not pulling their weight. Some attributes get dropped; others get justified; the attribute set stays bounded.
- Critical attributes: standard semconv.: The standard semantic conventions (HTTP, RPC, K8s, Database) provide universal value. These attributes are not subject to the trim; they are part of the foundation.
- Custom attributes: only the ones used in queries.: Beyond the standard semconv, custom attributes must justify themselves. If the attribute is queried, it stays; if not, it goes. The discipline is data-driven.
- Per-service budget allocation.: Different services have different attribute needs. The budget can vary; the discipline applies. Each service team owns their budget; the platform team enforces.
The budget is the discipline. Without it, attributes accumulate; with it, the cost stays bounded.
Trim periodically
The attribute set should be reviewed and trimmed periodically. Attributes accumulate over time; the trim keeps the set healthy.
- Quarterly: review attribute usage.: Once per quarter, the team reviews which attributes are actually used. The review produces a list of unused attributes; remediation removes them.
- Attributes not queried in 90 days are candidates for removal.: The 90-day window catches the truly unused. Some attributes might be queried only quarterly (compliance reports); the review accommodates these.
- Most teams find 30 to 50% reduction is safe.: When a team first applies the discipline, significant reduction is typical. The accumulated attributes include many that were added speculatively and never queried; removal is straightforward.
- Track the savings.: The cost reduction from each trim is tracked. The savings demonstrate the value; future trims have data to support them; the discipline is reinforced.
- New attributes have justification.: The discipline extends to new attributes. Adding a new attribute requires articulating why; the speculative additions are caught at PR review; the budget is preserved.
Trace attribute cost model is one of those observability cost disciplines that pays off proportionally to trace volume. Nova AI Ops integrates with tracing platforms and cost data, surfaces attribute usage patterns, and produces the per-attribute trim queue that drives the quarterly cleanup.