An Incident Runbook for AI Agents: What to Do When the Agent Does Something Wrong
November 11, 2026 · 6 min read · Etzal Group Team
At 21:40 on a Friday, an AI agent that replies to supplier emails begins confirming purchase orders that nobody approved. It has misread a change in the approval field. By the time someone notices on Monday, sixty confirmations have gone out. The cost of the error is modest. The cost of the sixty hours is not. Every one of those hours existed because nobody had decided, in advance, who watches the agent, who can stop it and how. An incident runbook for AI agents is that decision, written down.
Why AI agents need their own runbook
A conventional software fault usually stops something: a page fails, a job crashes. An agent fault often keeps going, fluently. The output looks normal. The agent may repeat the mistake hundreds of times, and each repetition is an action in the real world: a message sent, a record changed, a payment prepared.
Three features make these incidents different:
- Plausibility. Wrong output looks like right output.
- Speed and scale. One wrong rule applies to every case at once.
- Unclear cause. The trigger may be a changed input, a changed model version, a changed document in its knowledge base or an instruction hidden in content it read.
Before any incident: five things that must exist
1. A kill switch. One step that stops the agent acting: a setting, a feature flag, a disabled credential. Documented, tested, usable by several named people without a deployment. Prefer two levels: "draft only", where the agent prepares and people approve, and "off".
2. Logs. For every run: the input, the instructions and model version in force, the tools called with their arguments, the output, and what was sent or changed. Without these, the investigation is guesswork.
3. An owner. A named person responsible for the agent, and a deputy.
4. A way to detect. Alerts on unusual volume, on error rates, on actions near limits, on customer replies containing words like "wrong" or "why". And a channel where staff can report "the agent did something odd" in ten seconds.
5. Limits. Ceilings on what the agent may do without approval: value, number of actions per hour, classes of recipient. Limits turn a possible disaster into a bounded one.
Severity levels
Three are enough.
| Level | Definition | Response |
|---|---|---|
| 1, Critical | Harm to people outside the company, money moved, personal data exposed, or legal exposure | Stop the agent now, call the owner and management, start the full runbook |
| 2, Major | Wrong output or action reached customers or systems, limited and reversible | Switch to draft mode, owner investigates the same day |
| 3, Minor | Wrong output caught before it left, or cosmetic errors | Log it, fix in normal work, review weekly |
When in doubt, treat it as the higher level. It is cheap to downgrade.
The first hour
1. Stop or restrict. For level 1, off. For level 2, draft only. Do this before understanding the cause. Record the time.
2. Assign one incident lead. One person decides and coordinates. Others report to them.
3. Establish scope. From the logs: when did it start, how many runs, which customers, records or transactions were touched? Start with the earliest known bad output and search backwards for earlier ones.
4. Contain the effects. Recall or correct messages where possible. Reverse or freeze transactions. Revoke credentials the agent used if misuse is suspected. Flag affected records.
5. Preserve evidence. Export the logs, the instructions, the configuration, the knowledge documents and the model version as they were. Do not fix anything in place before a copy exists.
6. Notify internally. Management, the affected department, legal or compliance where relevant. A short factual note: what happened, since when, how many affected, what has been stopped, when the next update comes.
7. Decide on external communication. Who needs to hear from you, by when, and from whom.
Communicating with those affected
- Be early. People forgive an error more easily than learning about it from someone else.
- Be specific. What was wrong, what the correct position is, what they need to do, if anything.
- Take responsibility as the company. "Our system sent you an incorrect confirmation" is acceptable. "The AI made a mistake" as an excuse is not.
- Give a human contact.
- Do not speculate about causes you have not confirmed.
Where personal data is involved, the person named in the runbook assesses whether legal notification duties apply and within what time. Have that name and the adviser's number written in the runbook now.
Finding the cause
Work through the usual suspects in order:
- Did an input change? A new field, a new format, a new kind of request.
- Did the knowledge change? A document updated, added or left stale.
- Did the instructions change? A prompt edited, a rule added.
- Did the model change? A new version from the provider can alter behaviour without any change on your side.
- Did a tool change? An interface that returns something different.
- Was the agent manipulated? Content it read, such as an email or a web page, containing instructions it followed.
- Was this always possible and simply had not happened yet?
Reproduce the failure with the preserved input. A cause you cannot reproduce is a hypothesis.
Before switching back on
- The cause is identified and reproduced.
- A fix is in place and the failing case now passes.
- The failing case, and variations of it, are added to the test set that runs before every change.
- A guard exists that would have limited the damage: a limit, an approval step, an alert.
- The owner and the affected department agree to restart.
- Restart in draft mode first, then at low volume, with someone watching.
The review
Within a week, a one-page record:
- What happened, in plain words, with a timeline.
- Impact: who and what was affected, and the cost.
- Detection: how it was found, and how long after it started.
- Cause.
- What went well in the response.
- What was slow or missing.
- Actions, each with an owner and a date.
Keep it blameless. The question is what in the system allowed this, not who is at fault. People who fear blame hide near misses, and near misses are the cheapest warnings you will get.
Pay particular attention to the time between start and detection. Shortening that interval is usually worth more than any other improvement.
Practise once
Run a thirty-minute exercise each half year: "The agent has sent wrong prices to forty customers. Go." See whether people can find the kill switch, the logs and the contact list. Exercises reveal the missing password and the person on leave while it is still harmless.
Where this fits
A runbook is one part of operating agents responsibly, with limits, testing and monitoring. We build these into the agents we deliver and help clients write them for systems already running, as part of AI Consulting. If you have an agent in production and no written answer to "who stops it?", contact us.
Summary
Before any incident, have a tested kill switch, complete logs, a named owner, detection and limits. Define three severity levels. In the first hour, stop or restrict, appoint a lead, establish scope, contain, preserve evidence and notify. Communicate early and specifically. Find and reproduce the cause, add the case to your tests, restart in stages, and write a blameless review. Then rehearse it.
Frequently asked questions
What counts as an AI agent incident?
Any event where an agent's output or action caused, or could have caused, harm: wrong information sent to customers, an action taken without authority, personal data exposed, money moved incorrectly, or the agent behaving outside its instructions. Near misses caught before harm count too, because they show what will happen next time.
Who should be able to stop an AI agent?
More than one person, at any hour. At minimum the system owner, the on-call engineer and the operations lead of the team that uses it. Stopping must be a single documented step that needs no developer and no deployment. If only one person knows how, the agent is not ready for production.
Do we have to notify anyone when an AI agent leaks personal data?
Personal data laws in many jurisdictions, including Indonesia and Singapore, set obligations to notify the regulator and affected individuals after certain data breaches, with time limits. Whether and when a duty applies depends on the facts. The runbook should name the person who makes that assessment and the adviser they call.
Related service
WhatsApp AI Chatbot
We build, train, and run a WhatsApp AI agent that answers customers 24/7, qualifies leads, and follows up.
