Actionable vs Informational Alerts
Some alerts page; others go to dashboards. The distinction.
The rule
If a human cannot do something useful within 5 minutes of receiving the alert, it is informational, not actionable. Informational signals belong on dashboards or in Slack channels. Actionable signals page.
Pages must include a runbook link, a clear owner, and a measurable customer-impact statement. If any of those are missing, the alert is unfinished work.
Pages without action are the largest source of on-call burnout. Treat the page channel as a budget, not a default.
Informational signals
Capacity warnings 30 days from a ceiling, slow query logs, deprecation notices, and security tool drift reports. None require an immediate human response.
Route them to a Slack channel with a daily summary. Use Grafana or Datadog dashboards for trend visibility, not PagerDuty.
Periodic review wins here. Walk the dashboard at the weekly ops meeting; act when trends cross a threshold, not when the reading is taken.
Actionable signals
Customer-facing latency above SLO, error rates spiking, queue depth growing without bound, replication lag breaking RPO. The on-call can act in minutes.
Tie each actionable alert to a single primary signal. Compound conditions like "errors AND latency AND traffic above 50%" reduce false pages without losing coverage.
Validate yearly that each actionable alert has been useful. Alerts that fire and resolve themselves three times running should be downgraded.
Operational discipline
Track page-to-action ratio. Below 70% means you are paging too often. Above 95% means you are likely under-alerted on real problems.
Run a monthly noise budget. Anything over 5 informational pages per shift gets a JIRA ticket against the owning team to retune or delete.
Make deletion easy. New on-call should not feel they need permission to silence a flapping check; let them silence then file a ticket.
How to apply this week
Audit the last 30 days of pages. Count how many resulted in a real customer fix. Demote anything below 50% to a Slack channel.
Add a runbook URL field to every alert template. Block merges of alert configs without one.
Make the on-call rotation channel publish a weekly noise score. Visibility forces tuning faster than any policy memo.