Agentic SRE: Where AI Meets Operations
AIOps gave humans better dashboards. Agentic SRE replaces the human in the loop for routine incidents. Here is the architectural shift and where it actually works.
The architectural shift
Traditional SRE is human-led: alerts page humans; humans investigate; humans remediate. Agentic SRE introduces AI agents into that loop, not replacing humans, but handling the routine layers so humans focus on novel and high-stakes work. The shift is from "tools that humans use" to "agents that do work, supervised by humans".
The motivation. Modern systems generate too many signals for humans to investigate manually. Most alerts are routine, disk space, common deploy failures, known recurring issues. Humans triaging routine alerts is a misallocation of expensive attention. Agents handle the routine; humans handle what's novel.
The maturity curve. By 2026, agentic SRE is in production at several major operators. The technology works; the organisational adoption is the harder part. Teams that have committed to the model see meaningful reductions in MTTR and on-call burden; teams that haven't are still in pilot.
The honest framing. Agentic SRE doesn't eliminate human oncall, it changes its nature. Humans handle the harder, less common, higher-stakes incidents. The total volume of human attention drops; the per-incident attention increases. This is usually a good trade.
The risks. Agents that act incorrectly cause incidents. Agents that exhaust budget burn money. Agents that hide problems from humans erode trust. The risks are real; the architecture must address them or agentic SRE fails in production.
Four-layer architecture
The architectural pattern that works in production:
- Sensing layer, agents read metrics, logs, traces, alerts. Same data humans see.
- Reasoning layer, agents form hypotheses about what's happening. "DB is slow because of long-running query X."
- Action layer, agents execute remediations. Restart services, kill queries, scale resources, page humans for harder cases.
- Audit layer, every action is logged with full context. Humans review; agents learn from feedback.
The sensing-layer details. Standard observability inputs (Prometheus, OpenTelemetry, log aggregators). Agents query the same systems humans use. The architectural choice: should agents have full access or curated views? Most teams give full access with rate limiting; the alternative produces blind agents.
The reasoning-layer details. The agent's "thinking", what's wrong, why, what to do about it. Modern LLMs handle this well for routine cases. The agent's reasoning should be exposed to humans; opaque reasoning erodes trust quickly.
The action-layer details. The agent acts via the same APIs operators use, kubectl, AWS CLI, internal APIs. Permissions matter: what's the agent allowed to do? Most production deployments scope tightly: restart services in defined namespaces; not delete databases; not modify IAM.
The audit-layer details. Every agent action is logged: what was the input state, what reasoning, what action, what result. Logs enable post-incident review and continuous improvement. Without audit, agents are unaccountable; accountability is what makes humans trust them.
Autonomy in production
The honest answer: most production deployments give agents narrow autonomy. They can do well-understood remediations (restart pods, clear caches, kill rogue queries). They escalate to humans for anything novel or high-stakes. The autonomy boundary moves with track-record, agents earn more autonomy by demonstrating reliability.
The narrow-autonomy pattern. Define a list of approved actions. Agent can take any approved action without asking; takes anything else only with human approval. Approved-action list grows over time as agents prove reliable on each. Conservative starting; widening over months and quarters.
The escalation rules. When the agent doesn't know what to do, when it's about to take a high-stakes action, when its confidence is low, page a human. The escalation criteria should be explicit: "if pod restart count exceeds 3 in 5 minutes, escalate". Agents that escalate appropriately earn human trust faster.
The trust-building cycle. Agent acts; human reviews; agent learns from feedback. Over time, the agent's actions match what the human would have done. The agent earns autonomy. The cycle is months, not weeks; building trust is a deliberate process.
The "agent goes rogue" mitigation. Hard caps on agent actions per time window. "No more than N actions per minute." "No more than M total per day." When caps hit, agent stops and escalates. The caps prevent runaway scenarios; cost a small amount of speed for major safety.
Where agents win
The clear wins:
- Triage, agents read incoming alerts, gather context, summarise for humans. The "wake-up read" goes from 10 minutes to 30 seconds.
- Routine remediation, disk space, OOM restarts, transient errors. Hundreds of these per week; agents handle them while humans focus elsewhere.
- Postmortem drafting, agents pull timeline, logs, related incidents. Humans edit and reason; the heavy lifting of "compile the data" is automated.
- Runbook execution, agents follow runbooks for known incidents. Humans wrote the runbooks; agents execute reliably without typos.
The triage win. Routine alerts arrive constantly. Without agents, every one is a context-switch for an oncall engineer. With agents, the agent gathers context (what's the metric, what's recent, what's similar in history) and presents a summary. The engineer's first look is informed; decisions are faster.
The remediation win. Routine remediations have known fixes. Without agents, an engineer types kubectl rollout restart at 3am. With agents, the agent does it; the engineer is alerted only if the routine fix fails. The volume of "you were paged but you didn't have to do anything" alerts drops substantially.
The postmortem win. Postmortem authoring is time-consuming because the timeline data is scattered. Agents pull timelines from logs, deployments, alerts; populate templates; draft narratives. Humans edit for accuracy and reasoning. The hours-per-postmortem drops by 60-70%.
The runbook execution win. Runbooks describe responses; humans execute them. Humans make typos, miss steps, run things in wrong order. Agents execute runbooks deterministically. The reliability improvement is meaningful for incident response.
Common antipatterns
Giving agents broad autonomy on day one. Trust must be earned. Start narrow; widen after track record.
Opaque agent reasoning. Humans need to see why the agent decided. Without transparency, trust collapses on the first surprise.
No audit logs. Post-incident review is impossible. Always log; always make logs human-readable.
Skipping human review of agent actions. Even reliable agents drift. Periodic review catches drift before it causes incidents.
What to do this week
Three moves. (1) For one routine remediation that you do manually frequently, pilot agentic handling. The smallest successful pilot makes the case for widening. (2) Define the autonomy boundary explicitly. "Agent can do these actions without approval; not these." Without explicit boundary, you end up with implicit infinite autonomy. (3) Build the audit log review process. Spend 30 minutes weekly reviewing agent actions; learn what to add to the approved list and what to remove.