Alert Throttling During Active Incidents
During active incidents, related alerts are noise.
Why throttle during incidents
During a major incident, related alerts fire continuously. The on-call already knows; additional pages add noise without information.
Throttling caps the page rate per service per incident. Typical rule: max 1 page per 15 minutes per service while incident is open.
The first page wakes the human; the next 50 are paperwork.
How to throttle
PagerDuty event orchestration supports rate limiting per service. Opsgenie has alert deduplication policies. Nova AI Ops groups by incident.
Throttle by service first, then by team. Cross-team throttling can hide unrelated incidents.
Always log throttled alerts. The on-call should be able to see "these 14 alerts fired but were throttled during incident X".
When not to throttle
Security alerts. A breach during an outage is more dangerous, not less.
Data-loss alerts. Replication failures and backup misses must always page.
New, unrelated services. Throttle scoped to the incident, not blanket.
After the incident
Resume normal alerting within 5 minutes of the resolve. A 30-minute throttle window after resolve hides re-occurring issues.
Audit throttled alerts in the postmortem. Sometimes a throttled signal pointed to a separate incident that was missed.
If throttling fires more than weekly, your alert volume during incidents is too high. Tune the underlying signals, not the throttle window.
Apply this quarter
Configure rate limiting on your top 5 services. Cap at 1 page per 15 minutes per service while an incident is active.
Test in a game day. Trigger a fake incident, fire 20 alerts, confirm only one pages.
Review monthly. Throttling that suppresses real incidents is worse than no throttling.