First Tempo Install

Tracing backend.

Overview

Your first Grafana Tempo install is the moment trace storage becomes object-storage-backed and inexpensive. Tempo stores traces in S3/GCS/Azure Blob; the cost model changes from "premium retention" to "near-S3 storage cost," which changes what the team can afford to keep.

• Object-storage backed. Traces in S3/GCS/Azure Blob; the cheap-retention story; weeks of traces fit the budget.
• OTel-native. Receives OTel traces directly; matches modern instrumentation; vendor-neutral on the SDK side.
• Grafana integration. First-class Grafana data source; the trace surface lives next to metrics and logs.
• TraceQL plus service graph. Query language for traces; auto-derived dependency graph; the investigation surface for cross-service work.

The approach

The practical approach: monolithic mode to start, S3 backend for cheap retention, OTel ingest, tail-based sampling for value. The team’s discipline produces stable Tempo without the operational complexity of distributed mode.

• Monolithic mode to start. Single binary for small deployments; the right shape until volume justifies distributed mode.
• S3 backend. Cheap object storage; weeks of traces at near-zero unit cost; the retention story is the product.
• OTel ingest. Receive traces from OTel SDKs and Collector; vendor-neutral; matches modern stacks.
• Sampling. Tail-sampling for value preservation; keep the slow traces, sample the fast; matches volume to budget.
• Document the install. Per-cluster configuration committed to the repo; supports investigation and rebuild.

Why this compounds

Tempo mastery compounds across services. Each instrumented service grows the team’s tracing surface; cost-per-trace stays near-zero as adoption scales.

• Faster cross-service investigation. Traces show the full request path; the bottleneck is identified by the trace, not by guess.
• Better cost efficiency. S3 storage produces cheap retention; the bill scales linearly with volume, not exponentially.
• Reusable patterns. Standard ingest and query patterns; new services inherit the muscle memory.
• Institutional knowledge. Each trace teaches the system; the team’s observability muscle grows.

The first Tempo install is an infrastructure investment that pays off across years. Nova AI Ops integrates with tracing telemetry, surfaces patterns, and supports the team’s observability discipline.