Alert Scope Creep Prevention

Alerts that get tweaked drift from purpose. Prevent.

The problem

Alerts created with a clear purpose drift over time. A new condition is added, then another, until the alert fires on a different problem than originally intended.

Six months later, the on-call sees the alert fire and has no idea what they should do. The original purpose is lost.

Scope creep is silent. It happens through small PRs, each defensible in isolation.

How it happens

An incident teaches you that condition X also matters. You add it to an existing alert instead of creating a new one.

A team member who didn't write the alert tweaks the threshold to fix a single false positive. The alert now fires for a different reason than it was designed for.

Aggregate operators (max, p99) get added incrementally, then nobody knows which signal is actually being measured.

Prevention

Every alert has a one-line purpose statement. "Detect customer-facing checkout latency above SLO". This is in the alert payload.

Code review: any change to the alert logic must explain whether the purpose statement still holds.

Quarterly alert review. Walk the catalog with the owning team. Anything where the purpose has drifted gets split into two alerts or deleted.

Split vs extend

Default to splitting. Two alerts with clear purposes are easier to operate than one alert with two purposes.

Extend only when the underlying signal is genuinely the same. "p99 from Prometheus" plus "p99 from Datadog APM" is one alert with redundancy, not two purposes.

If a single alert needs two runbooks, it's two alerts.

Apply this quarter

Audit your top 10 paging alerts. Confirm each has a single, clear purpose.

Split any alert with two purposes. Migrate carefully; don't delete the original until the new alerts have run for 2 weeks.

Add the purpose-statement field to your alert template. Reject new alerts without one.