Alert vs Dashboard Decision
Some signals belong on dashboards, not in alerts.
The decision rule
Alert if: customer impact is happening or imminent, time-sensitive action is required, and someone needs to act now.
Dashboard if: trending data, aggregate metrics, situational awareness, or post-hoc analysis. The data informs decisions but doesn't demand immediate action.
Mixing the two creates fatigue. Dashboards full of pageable signals get ignored. Pages that should have been dashboards burn out the on-call.
Strict criteria for alerts
Customer impact, real or imminent. A signal with no customer connection (CPU at 80%, RAM at 70%) is a dashboard, not a page.
Action exists. An alert without a runbook is a notification of helplessness. Either find an action or move it to a dashboard.
Time-sensitive. If the action can wait until business hours, the alert can wait. Otherwise it pages now.
Dashboard criteria
Trends and aggregates. Week-over-week, month-over-month metrics. Capacity planning data. SLO burn-down charts.
Operational awareness. The on-call checks at start of shift. During incidents, dashboards inform; they don't drive paging.
Stakeholder reports. Business metrics, customer counts, revenue dashboards. Pages would be wrong here; the audience is decision makers.
Converting between them
Frequently-firing alerts that operators dismiss without action are dashboard candidates. Track per-alert action rate; below 50% means it should not be an alert.
Dashboard panels that surface real problems people only see in postmortems are alert candidates. Convert when the pattern repeats.
Quarterly review of both directions. Each conversion is a small win; cumulative effect is significant alert quality.
Anti-patterns
Dashboards full of red panels nobody investigates. Dashboards are not alerts; the visual urgency creates anxiety without action.
Alerts that exist for reassurance ('alert if too quiet') without clear meaning. Either define what 'too quiet' means and what to do, or remove.
Both alert and dashboard for the same signal. Pick one based on the action; if both exist, ensure they have different audiences and clear ownership.