Alert History Export

Alert history is data. Export it for analysis.

Why export alert history

Alerting tools keep 30-90 days of history by default. That is too short for trend analysis, postmortem reviews, or auditing.

Export to long-term storage (BigQuery, Snowflake, S3 + Athena). 18-24 months retention is the right floor.

The export is the foundation for the cleanup ritual, the on-call survey, and the noise budget.

How to wire the export

PagerDuty: webhooks fire on incident events. Pipe to a Lambda or Cloud Function, write to a warehouse table.

Datadog and Prometheus Alertmanager both support webhook receivers. Use the same Lambda pattern.

Schema: incident_id, alert_name, fired_at, acked_at, resolved_at, owner_team, severity, labels (JSON).

Retention and access

18 months minimum. 24 months covers full year-over-year analysis with one rollover.

Encrypt at rest. Strip PII (user IDs, IPs) at ingest using a deny-list on label names.

Restrict access to SREs and engineering leads. Alert history reveals who burns out and which teams are noisy; treat it like HR data.

Queries that pay back the work

Top noisy alerts. Drives the quarterly cleanup.

Time-to-ack and time-to-resolve per team. Drives rotation rebalancing.

Alerts that fired during deploy windows. Catches deploy-induced noise that should be silenced or fixed.

Build vs buy

PagerDuty Insights and Datadog Watchdog cover the basics. Use them until you outgrow the queries they support.

Build a custom warehouse only when you need cross-tool analysis or labels they don't expose.

Don't build the export to look smart. Build it because the queries you cannot run today are worth running.