Logs Cost Optimization

Log volume drives cost. Tune.

Overview

Log volume drives observability cost more than any other lever. Datadog, Splunk, and CloudWatch all bill on ingest and retention; selective logging produces real value at controlled cost while shotgun-debug logging produces a $50k surprise on the next invoice. The discipline is treating log lines as data the team pays for, not as free emissions.

• Volume per service. Track ingest GB by service. Reality lands on the dashboard rather than the bill.
• Per-tier log level. Tier 0 services log at INFO; tier 3 batch jobs log at WARN. Priority matches verbosity.
• Noisy-log drops. Quarterly review of top-talker patterns. Drop or sample at the agent before they reach the platform.
• Cold tier for old logs. S3 + Athena (or equivalent) after 30 days. Hot retention stays small; investigations after 30 days remain possible.

The approach

Tune the level by tier, drop the noisy talkers, tier old logs to cold storage, audit quarterly, document the policy. The discipline scales with the team because the rules live in code (agent config) rather than tribal knowledge.

• Per-tier log level. Tier 0 INFO, tier 1 INFO with sampling, tier 2 WARN, batch WARN. Priority matches verbosity.
• Drop noisy logs at the agent. Filter known-spam patterns before ingest. Quarterly review keeps the rules current.
• Cold tier after 30 days. Old logs roll to S3 + Athena (or equivalent). Hot retention stays cheap; investigations stay possible.
• Quarterly audit plus documented policy. Per-team log policy in writing; quarterly review catches drift between policy and reality.

Why this compounds

Each correctly scoped service produces ongoing savings on every subsequent invoice. The team's observability culture matures from "log everything just in case" to "log what we will actually read." By year two cost-aware logging is the default and new services ship with policy-conformant config on day one.

• Better cost efficiency. Volume matched to workload value. The bill stops surprising finance.
• Better operational fit. Hot logs are the ones investigators actually read. Signal-to-noise improves.
• Engineering culture matures. Cost awareness becomes part of operations rather than a quarterly fire drill.
• Year-one investment, year-two habit. First audit sets the patterns; by year two, every new service ships with policy-conformant logging.