Alert Template Discipline

Alert messages should be informative. The template.

Why templates

Hand-rolled alerts vary in quality. Templates encode the team standard: required labels, runbook link, dashboard link, severity rules.

Templates reduce review burden. Reviewer checks the template was used, not whether each field is correct.

Templates make audits cheap. Linter on the template fields is a 1-line check.

What a template includes

Required labels: owner_team, severity, runbook_url, dashboard_url, service, environment.

Required annotations: summary (under 80 chars), description (under 500 chars), impact (one sentence).

Default routing rules. Sev1 to PagerDuty, sev2 to PagerDuty during business hours, sev3 to Slack-only.

Per-category variants

Latency alert template: takes service name, threshold, time window. Generates the SLO burn-rate query.

Error-rate template: takes service name, threshold, comparison window.

Saturation template: takes resource, threshold, sustained-duration.

How to enforce the template

Helm chart, Terraform module, or Jsonnet library that wraps the template. Authors call the function; they cannot bypass.

Linter rejects raw alert config that doesn't go through the wrapper.

PR template asks: which alert template did you use, and why.

Start small

Pick the 3 most common alert shapes. Build templates for those first.

Don't try to template every edge case on day one. Custom alerts are still allowed; they just need explicit approval.

Template adoption is a leading indicator. If less than 80% of alerts use a template after 6 months, the templates are too rigid.