Log Aggregation Storage Tradeoffs

Hot, warm, cold storage tiers. The transition policies, the cost ratios, and the queries that get fast or slow at each tier.

Hot tier: 7 days

Log aggregation storage trade-offs are the cost-vs-access decisions that determine the team's logging bill. Hot, warm, and cold tiers each serve different access patterns; matching the data to the right tier produces sustainable cost.

What hot tier provides:

• Indexed, queryable in seconds.: The hot tier's data is fully indexed. Queries return in seconds; the iteration loop for active debugging is preserved.
• Most expensive per GB.: The cost reflects the capability. Indexed storage with low-latency query is more expensive than cold storage; the team pays for the access pattern.
• Used for active debugging.: Recent incidents, current investigations, real-time monitoring all hit the hot tier. The data needs to be accessible quickly; the cost is justified by the use.
• Right-size.: The hot tier should match the team's actual debug window. Most teams need 7 days; some need 14; some need 3. Sizing past the actual need wastes spend.
• Hot tier should fit the team's debug-window.: Beyond the window, the data is rarely queried. Keeping it in hot tier costs money without producing value; the data should move to warm.
• Beyond that is wasted spend.: The hot tier's premium is justified only for actively-queried data. Old data in hot tier is paying premium for cold-tier access patterns.

The hot tier is for active work. Sizing it to actual need is the discipline that controls cost.

Warm tier: 7-90 days

The warm tier holds data between active and archival. Less indexing, slower queries, lower cost. The trade-off matches the access pattern: occasional queries on older data.

• Less indexed.: The warm tier's index is reduced. Some fields might not be indexed; some queries might require scans rather than lookups. The reduced indexing reduces storage cost.
• Queryable in minutes.: Queries against warm data return in minutes rather than seconds. The latency is acceptable for postmortem investigation; the cost reduction is significant.
• 5x cheaper than hot.: The warm tier's cost is roughly 5x cheaper than hot. The savings are substantial; the access pattern's reduced demands justify the savings.
• Useful for postmortems.: Postmortem investigation typically looks at recent-but-not-current data. The warm tier serves this access pattern; the slower query is acceptable.
• Trend analysis.: Looking at trends over weeks uses warm tier data. The trends are stable; the queries can take minutes; the cost is appropriate.
• Slower queries are acceptable here.: The team accepts the slower queries. The use case tolerates them; the savings are worth the trade-off.

The warm tier is the middle ground. It serves the in-between access patterns at appropriate cost.

Cold tier: 90+ days

The cold tier holds long-term data. Object storage, slow queries, very low cost. The tier's primary use is compliance retention; rare deep investigations also touch it.

• Object storage.: The cold tier uses S3 or equivalent. The storage cost is per-GB at object-storage rates; the storage is essentially free at low volumes.
• Queryable in tens of minutes via batch jobs.: Cold queries are slow. They typically run as batch jobs rather than interactive queries. The latency is high; the cost is low.
• Used for compliance.: Many compliance regimes require long-term retention. The cold tier holds the data for the required duration; the queries are rare; the cost is bounded.
• And rare deep investigations.: Some investigations need very old data. The cold tier accommodates these; the slow queries are acceptable for the rare use.
• Cheap.: The per-GB cost is much lower than hot or warm. Long retention is feasible; the storage cost stays bounded even at large volumes.
• Cold queries are not for daily use.: The team accepts that cold tier queries take long. The use cases that need cold data are rare; the access pattern matches the cost characteristics.

Log aggregation storage trade-offs are one of those FinOps disciplines that pays off proportionally to log volume. Nova AI Ops integrates with logging platforms and storage, surfaces tier-distribution patterns, and produces the cost optimization recommendations that match data to the right tier.