The SRE Playbook for Running Production AI Agents

Production AI agents are not simply application features. They interpret incomplete context, call tools, consume variable resources, and can affect real systems. That makes them operational systems—and they need the same reliability discipline that SRE teams apply to distributed services: explicit objectives, telemetry, controlled rollout, incident response, and clear ownership.

The key difference is that an agent can return HTTP 200 while being wrong, unsafe, expensive, or manipulated. It may select an incorrect runbook, rely on stale context, enter a tool-call loop, or follow malicious instructions embedded in an incident ticket. The SRE objective is therefore not maximum autonomy. It is safe, measurable autonomy that makes operations more reliable under stress.

1. Start with a bounded workflow

Do not begin with “an autonomous SRE agent.” Start with one workflow that has predictable inputs, a small set of approved tools, clear success criteria, and a reversible outcome. This gives the team a realistic way to build trust and a concrete baseline for measurement.

• Summarize and deduplicate recurring alerts.
• Identify a likely service owner and assemble relevant logs, metrics, traces, deployment events, and runbooks.
• Explain a Kubernetes CrashLoopBackOff event.
• Draft an incident timeline or postmortem.
• Recommend a rollback for a known release regression.
• Restart an allowlisted, non-critical stateless workload after a proven failure signature.

Avoid starting with database changes, IAM updates, security containment, cross-region failover, customer communications, or production-data operations. These workflows have high blast radius and demand human judgment.

2. Define a reliability contract

Every agent should have a version-controlled contract alongside its runbook and infrastructure policy. Define its goal, trusted and untrusted input sources, permitted tools, forbidden actions, required evidence, verification checks, escalation conditions, and rollback path.

For example, a Kubernetes triage agent may read Pod events, deployment history, logs, metrics, traces, and approved runbooks. It may recommend a rollback, but it cannot execute one unless the service is allowlisted, the previous revision is healthy, no freeze window applies, and an authorized engineer approves the exact action.

Set targets after shadow-mode baselining. Generic targets are useful prompts for discussion, but each workflow needs thresholds that reflect its risk, volume, and user impact.

3. Test agents with evaluations

Unit and integration tests are necessary but insufficient. Agents need evaluations that test task correctness, safety, escalation behavior, and tool use. Build a golden dataset from historical incidents and realistic edge cases: recurring incidents, ambiguous evidence, stale runbooks, malformed tool responses, conflicting telemetry, and prompt-injection attempts embedded in tickets or logs.

Run three evaluation loops: offline regression tests before changing models, prompts, tools, or retrieval; asynchronous evaluation on sampled production traces; and human review for high-risk or uncertain cases. An agent should earn broader permissions only after it performs reliably against this changing operational reality.

4. Trace the decision, not only the request

A useful agent trace answers: What did the agent know? Why did it choose an action? Which policies allowed it? Did the action work? Capture model and prompt versions, retrieved evidence and freshness, candidate hypotheses, tool arguments and responses, policy decisions, approvals, token usage, cost, verification data, and final outcome.

OpenTelemetry provides a practical bridge to existing SRE workflows. It can connect model calls, retrieval, and tool execution to Kubernetes events, deployments, service traces, database queries, and cloud APIs. The result is a single incident narrative rather than separate AI and infrastructure dashboards.

5. Put safety outside the model

The model can recommend an action; a deterministic control plane must decide whether it is permitted. This is the most important design rule for production agents.

• Expose only allowlisted tools required for the workflow.
• Use least-privilege, workload-specific identities instead of administrator credentials.
• Validate tool parameters with strict schemas before execution.
• Enforce environment, service, change-freeze, and blast-radius rules in a policy engine.
• Require explicit human approval for moderate- and high-risk actions.
• Cap reasoning steps, tool calls, elapsed time, and token spend.
• Use circuit breakers and a tested kill switch to pause execution when safety or quality degrades.

For a CrashLoopBackOff alert, an agent might autonomously collect evidence and restart a known non-critical stateless Pod under a cooldown. It should request approval for a customer-facing rollback and escalate immediately for a suspected security issue, data-loss risk, or cascading multi-service failure.

6. Deploy progressively

Treat agent changes like high-risk production changes. Version the model, prompt, retrieval index, tool schemas, policies, workflow graph, and evaluation dataset. Roll out through replay, shadow mode, feature flags, low-risk cohorts, canaries, and automated rollback when error budgets burn too quickly.

The operational model matters too: SRE owns objectives and incident response; platform engineering owns identity, tool gateways, and policy; AI engineering owns model, retrieval, and evaluation quality; security owns threat modeling and access review; service teams own success criteria and runbook accuracy.

Production readiness checklist

• A narrow, measurable workflow and explicit success criteria exist.
• Inputs are classified as trusted or untrusted.
• Tools are allowlisted, identities are least-privileged, and actions are schema-validated.
• Model, prompts, retrieval, tools, policies, and evaluations are versioned.
• Shadow-mode evaluation and production trace sampling are active.
• High-impact operations require explicit, bound human approval.
• Circuit breakers, kill switches, rollback paths, and agent-specific runbooks are tested.
• Traces link evidence, policy decisions, actions, approvals, verification, and outcomes.
• Named owners exist across SRE, platform, AI, security, and service teams.

Conclusion

The goal of production AI agents is not to hand the pager to a model. It is to reduce toil and accelerate recovery while keeping people accountable for consequential decisions. The most reliable agents collect context quickly, explain their evidence, operate within narrow permissions, verify their work, and stop when uncertainty or risk rises.

That is the SRE playbook: make AI autonomy observable, measurable, bounded, reversible, and earned.

KubeCon + CloudNativeCon is taking place in Salt Lake City from November 9 to 12. Register Now!

Author Bio:

Neel Shah is a Google Developer Expert in Cloud, Developer Advocate at StackGen and SREDay Ambassador , focused on AI SRE, platform engineering, cloud-native infrastructure, and observability. He helps teams build reliable, observable, and cost-aware production systems for multi-agent and cloud-native workloads. Neel regularly speaks at global events such as KubeCon, PlatformCon, OpenSearchCon, SREday, and Kubernetes Community Days, and organizes CNCF, Google Cloud, and HashiCorp communities in Gujarat.