Runbooks for AI agents: what to write before the pager goes off

Agents fail in ways ordinary services do not: a changed prompt, a model update or a poisoned document can alter behaviour without a single deploy. The first response still looks like any incident, but the on-call engineer needs agent-specific steps, written in advance.

Why write it down

The Google SRE book reports that thinking through and recording best practices ahead of time in a playbook produces roughly a threefold improvement in mean time to repair, compared with improvising. That figure is Google's experience with its own services, not a promise for yours, but the direction is clear.

Stop first, explain later

The costliest mistake in an agent incident is investigating while the agent keeps acting. The runbook should put containment before diagnosis: pause the agent or cap its rate, and revoke or narrow its credentials if it may be acting on injected instructions.

What every runbook contains

• The symptom and the alert that points here, with the dashboard link.
• Severity and who to page, so nobody debates it during the incident.
• Diagnosis steps with the exact command and the output you expect.
• Mitigation steps. Destructive ones are marked and say how to undo them.
• How to verify recovery, and when to roll back instead.
• A link to the postmortem template.

Agent-specific entries

• Roll back the last change to the prompt, model or configuration, not only the code.
• Find which tool calls the agent made, with which identity, using the audit trail.
• Decide which actions must be reviewed by a person before the agent resumes. Anthropic's guidance on agents notes that they can pause for human feedback at checkpoints.

Test it

A runbook nobody has run is a guess. Rehearse the kill switch and the rollback on a schedule, and fix the steps that no longer work.