Quick answerAn error budget for AI agent hallucination rate works the same way an SRE error budget works for uptime: pick an acceptable hallucination rate for a defined period (say, a rolling 30 days), track actual rate against it continuously, and tie concrete governance consequences to how much of the budget has been consumed — slow down or freeze new capability launches once you've burned a defined share of the budget, and treat staying comfortably under budget as license to ship faster, not just as a passive good outcome. This is a standing policy that governs release velocity over time, set before problems occur, not a one-off number calculated after an incident.
Why this is a different tool than a real-time circuit breaker
A self-imposed circuit breaker that automatically throttles high-risk actions the moment the agent's own error signals spike is a real-time, automated technical mechanism — it reacts to a specific spike, in the moment, without a human involved. An error budget policy operates on a completely different timescale and purpose: it's a periodic, human-governed decision framework that determines whether the team is allowed to ship new capabilities or take on more risk this month, based on a rolling accounting of hallucination rate against a pre-agreed allowance. A circuit breaker can be one of several mechanisms that helps you stay inside your error budget; the budget itself is the governance layer that decides what "acceptable" means and what happens as you approach the limit.
Setting the number is a business decision, not just a technical one
The acceptable hallucination rate isn't a purely technical judgment — it should be derived from what the business can actually absorb, using the same underlying math as translating hallucination rate into a dollar cost for the CFO: multiply a candidate rate by conversation volume and the average cost per erroneous interaction, and work backward to the rate that keeps expected cost within a tolerance the business has actually agreed to. Setting the budget this way makes it defensible in front of the CFO and the board, rather than an engineering-chosen number that nobody outside the team actually agreed was acceptable.
Define the burn-rate consequences before you need them
The value of an error budget comes from consequences that are pre-agreed, not negotiated in the moment when the budget is already under pressure. Define, in writing, what happens at defined burn thresholds: at 50% of the period's budget consumed, flag it for review; at 75%, pause new higher-risk capability launches until the rate recovers; at 100%, treat it as a standing incident requiring the same response cadence as any other production reliability failure, until the rate is back under control. Without these thresholds agreed in advance, a team under launch pressure will rationalize continuing to ship even as the budget burns, which defeats the entire purpose of having one.
Keep this separate from per-incident triage
Triaging whether a specific bad answer was a hallucination or a product bug is the reactive process for a single incident, run every time something goes wrong. The error budget policy sits above that process: individual triaged incidents feed the aggregate rate the budget tracks, but the budget itself is about the standing, period-level decision of how much cumulative risk the business has agreed to tolerate — a well-run triage process makes the input to the budget more accurate; it doesn't replace the need for the budget's forward-looking governance function.
FAQ
How is this different from just tracking hallucination rate as a metric? Tracking a metric with no pre-agreed consequences is observability, not governance — teams routinely watch a metric drift upward without changing behavior because nothing was agreed in advance about what should happen at a given threshold. An error budget only works because the consequences of burning it are decided ahead of time, not debated after the fact.
Should the budget period be the same length across all AI agent capabilities? Not necessarily — a newer, higher-risk capability might warrant a shorter budget period (weekly) so problems surface and get addressed faster, while a mature, stable capability can reasonably use a longer period (monthly or quarterly) without losing meaningful signal.
What if we consistently stay well under budget — should the number change? Yes, revisit it periodically. Consistently staying far under budget may mean the number was set too conservatively relative to what the business can actually tolerate, and tightening it (or using the headroom to justify shipping higher-risk capabilities faster) is a legitimate use of that data, not just a passive good sign.