Alert Action Distinction

Alerts that fire actions vs alerts that just notify. The pattern.

Two classes of alert

Action alerts demand a human response within minutes. Page someone, wake them up, expect a runbook.

Notification alerts inform but require no action right now. They land in Slack, email, or a ticket queue for the next business day.

Mixing the two is the root cause of fatigue. A pager that fires for FYI events trains responders to ignore it.

How to classify each rule

Ask the runbook question: does the responder do something specific in the next 15 minutes? If not, it's a notification, not a page.

Use severity tiers explicitly: Sev1 pages on-call, Sev2 opens a ticket, Sev3 emails the owning team. Map each alert to one tier at creation.

Reject rules without a runbook. No runbook means no action, which means no page.

Routing the two cleanly

Alertmanager receivers split by severity label. PagerDuty for sev1, Slack webhook for sev2, email for sev3. No rule sends to more than one tier.

Disable mobile push for the notification channel. The phone is reserved for action.

Ticket creation should be idempotent. Use the alert fingerprint as the ticket key to avoid duplicates during flapping.

Review cadence

Quarterly: scan the action tier and demote any alert that produced no remediation in 90 days. Promotion in the other direction is rare and needs a post-incident finding to back it up.

Track demotion rate as a noise indicator. A team demoting 20% of action alerts per quarter is signaling that initial classification is wrong.

Audit Slack-only alerts too. A notification that everyone ignores is just clutter. Delete it.

Default to notification

When in doubt, classify a new alert as notification. Promotion to action requires evidence of an incident missed by the lower tier.

This inverts the common reflex of paging on everything. Inverting it is the point.

Skip if your team has fewer than 30 alerts total. The classification overhead is bigger than the noise.