On-Call Runbook Quality

Quality scoring; refresh cadence.

Overview

On-call runbook quality measures and maintains runbook freshness. Runbook count is the easy metric; quality runbooks are what actually reduce MTTR. The library is only as useful as its worst-rated entry.

• Quality scoring plus refresh cadence. Per-runbook quality score; the score determines whether on-call trusts the runbook.
• Per-runbook freshness. Last-updated date per runbook; over 6 months is dated; over 12 months is suspect.
• Per-incident runbook usage. Per-incident the runbook used; the data shows which runbooks are actually consulted.
• Per-quarter refresh plus owner. Quarterly refresh catches drift; per-runbook owner supports accountability.

The approach

The practical approach: per-runbook owner named, quarterly refresh scheduled, per-incident usage tracked, documented policy. The team’s discipline produces fresh runbooks instead of a graveyard of stale documents.

• Per-runbook owner. Each runbook has a named owner; supports accountability; "everyone’s runbook" rots into "no one’s."
• Per-quarter refresh. Quarterly runbook refresh; the cadence catches drift before it becomes an incident-time discovery.
• Per-incident runbook usage. Track which runbooks were consulted per incident; the data prioritises the next refresh.
• Per-runbook freshness. Last-updated date visible; the on-call sees the freshness before trusting the runbook.
• Document the policy. Per-team runbook policy committed to the handbook; supports operational reviews.

Why this compounds

Runbook quality discipline compounds across rotations. Each fresh runbook reduces MTTR; the team’s on-call maturity grows; new joiners onboard faster on a trustworthy library.

• Better incident response. Fresh runbooks reduce MTTR; the on-call follows the runbook instead of investigating from scratch.
• Better onboarding. Fresh runbooks support velocity; new on-callers ramp on a current library, not archaeology.
• Better culture. Runbook quality signals that on-call matters; the team invests because the team uses the output.
• Institutional knowledge. Each runbook teaches operational patterns; the team’s collective on-call expertise grows.

Runbook quality discipline is an operational discipline that pays off across years. Nova AI Ops integrates with runbook telemetry, surfaces patterns, and supports the team’s on-call discipline.