Auto-Tuning Alert Thresholds with an Agent
Static thresholds rot. The agent that profiles each alert, proposes a new threshold, and lets you accept or reject the suggestion.
Why static thresholds rot
A threshold set at deploy time reflects the system's behaviour then. The system grows; the threshold does not. The alert becomes too sensitive or too lax.
Manual re-tuning is expensive: each engineer adjusts a few thresholds; the rest stay rotted. The result is the alert quality the team has today.
Auto-tuning solves this for high-volume alerts. Low-volume alerts still benefit from human attention; the agent does not handle every threshold.
Profile the alert
Pull the metric values over the last 90 days. Compute distribution: median, p90, p95, p99.
Cross-reference with the alert firing history. "This threshold fired N times; M were real incidents, K were false positives."
The signal-to-noise ratio guides the proposal. A threshold that fires often without real incidents is too sensitive; one that misses real incidents is too lax.
Propose, do not apply
The agent emits a proposed new threshold with justification: distribution data, hit-rate data, recommended new value.
Operators review the proposal. They accept, reject, or modify. The agent does not apply unilaterally because alert thresholds are policy decisions.
Acceptance is recorded. Future tunings learn from past acceptances: "this team prefers conservative thresholds" or "this team accepts aggressive ones."
How often to tune
Quarterly for most alerts. The system changes slowly; tuning more often is noise.
Monthly for alerts on rapidly-evolving services (new launches, scaling efforts).
Annually for stable, low-volume alerts. They rarely need changes; the agent flags only the ones that have drifted significantly.
Track the impact of tuning
Per-tuning: did the new threshold improve signal-to-noise? Compare 30 days before and after.
Aggregate: across all tuned alerts, the team's alert-to-incident ratio. Should improve; if not, the tuning agent needs work.
Operator satisfaction: a quarterly survey. The qualitative signal complements the quantitative metrics.