Connecting an Agent to GitHub for Runbook Updates

After every incident, the agent proposes a runbook update as a PR. The PR template, the review flow, and the metrics that prove this is actually compounding.

Runbook updates as PRs

After every incident the agent handled, propose a runbook update as a PR. The PR captures what was learned: new symptom, new diagnostic step, new remediation.

PRs are small: a few lines added, often one or two new sections. The reviewer approves in minutes.

Compounding effect: the runbook gets better; future agent runs handle more cases correctly.

PR template

Title: "[Agent] Update X runbook based on incident Y"

Body: incident summary, what the agent did, what was learned, what is being added to the runbook.

Files changed: the runbook file with the addition.

Tags: agent-generated, ready-for-review.

Review flow

Reviewer is the runbook owner. They approve, modify, or reject.

Most agent-generated PRs are approved with minor edits. Rejections happen when the agent misunderstood the incident; the rejection is logged and feeds back into agent improvement.

Approval merges; merge triggers a downstream sync to the runbook display tool.

Metrics that prove this works

Runbook update PRs filed per week: started at 0, target 5-10 in a healthy team.

Approval rate: target > 70%. Lower means the agent is misjudging; higher means the agent is too conservative.

Agent handle-rate of incidents covered by recent runbook updates: should improve over time. The proof of compounding.

Guardrails

Branch limit: at most 5 open PRs from the agent at once. Prevents flood.

Rate limit: at most 1 PR per runbook per week. Prevents churn on a single document.

Stale-PR cleanup: PRs older than 14 days without review are auto-closed. The agent re-files if the lesson is still relevant.