The Incident Rehearsal Quarterly Cadence

Practice your worst case quarterly. The format, the difficulty curve, and the gaps it surfaces in runbooks and team coordination.

Format

2-hour block. Pre-announced; the team knows it is coming but not what.

A senior engineer plays incident commander; injects scenarios; observes responses.

Real on-call tools, real channels, real escalation paths. Just a simulated incident.

Scenario examples

Database failover during peak traffic. Tests rollover playbooks.

Multi-region outage. Tests cross-region coordination.

Vendor API down. Tests degradation paths and customer comms.

Output

List of gaps: runbooks that were unclear, tools that did not work, processes that broke down.

Action items: each gap gets an owner and a fix-by date.

Drift caught early; real incidents go better because of the drill.