Alert Tuning Cadence: A Quarterly Discipline
Alert tuning has to be a recurring meeting on the calendar. Without the cadence, every team drifts back to noise.
Why cadence beats heroics
Alerts deteriorate predictably. New alerts get added; thresholds drift; teams change.
One quarterly meeting beats five heroic alert-cleanup days a year.
The four-hour quarterly agenda
- Hour 1: Top-10 alerts by volume. Decide retire/tune/keep for each.
- Hour 2: Pages-per-shift trends. Identify any team above 4/shift; intervene.
- Hour 3: Silence audit. Anything older than one quarter gets revisited.
- Hour 4: New service onboarding alert reviews; sign off or send back.
Artifacts each quarter produces
A list of retired alerts, tuned thresholds, and decisions made. Pinned in the team channel.
The next quarter starts from this artifact, not from scratch.
Annual deeper review
Annual: re-evaluate the SLOs that drive the alerts. SLOs that no longer match user expectations need updating before alerts can be tuned to them.
Without the annual SLO check, alert tuning is optimizing for the wrong target.
Antipatterns
- No cadence. Drift wins.
- Cadence without retirement. Volume only grows.
- Engineering manager runs the meeting alone. On-call must be present.
What to do this week
Three moves. (1) Apply this pattern to your noisiest alert. (2) Measure pages-per-shift before/after for one week. (3) Schedule the quarterly review so the discipline survives team turnover.