Quick answerAn AI agent's uptime metric needs to define at least three separate failure categories before it means anything: full outage (the agent is unreachable), functional degradation (the agent responds but with materially reduced capability, such as falling back to a smaller model or losing tool access), and quality degradation (the agent responds normally but with a measurably higher error or hallucination rate). Treating only the first category as downtime, the way traditional web-service SLAs do, will understate real customer impact for an AI agent by a wide margin.
Traditional uptime metrics were built for a different failure mode
A traditional web service is either serving requests or it isn't, which is why "uptime" as a binary up/down metric made sense for decades. An AI agent breaks that assumption. It can be fully reachable, responding within normal latency, and still be functionally useless or actively wrong, because the underlying model degraded, a tool integration silently failed, or a fallback model kicked in with materially worse capability. None of that shows up in a binary uptime check.
This is about building the underlying measurement definition, not the customer-facing page. Building the actual status page is a separate decision that only makes sense once you know what you're measuring and reporting on it.
The three-tier definition that actually reflects reality
Full outage: the agent cannot be reached at all, or returns errors instead of any response. This is the traditional definition and it still matters, but it's not the whole picture.
Functional degradation: the agent responds, but with a meaningfully reduced capability set, most commonly because a fallback model activated, a tool or data source it depends on is down, or a rate limit forced a reduced-context mode. Customers experience this as "it's working but it's worse," which traditional monitoring often misses entirely.
Quality degradation: the agent appears to be operating normally by every infrastructure metric, but its actual answer quality (measured against a golden eval set or sampled human review) has dropped. This is the hardest to detect in real time and the easiest to ignore, because nothing in your infrastructure dashboard will flag it.
From the team
We build production AI systems for startups.
LLM pipelines, RAG, and agent workflows that hold up under real traffic — not just in the demo.
Setting the actual thresholds
Full outage thresholds are straightforward: any period where requests fail or time out beyond your defined latency ceiling. Functional degradation needs a clear trigger, typically "fallback model active" or "tool call failure rate above X%" logged as a distinct signal, not folded into general uptime. Quality degradation needs a running eval sample, even a small one, checked on a fixed cadence, with a defined score drop that counts as degraded.
Why this matters even before you publish anything externally
Even if you never build a customer-facing status page, this methodology matters internally, because it's the only way to know whether your reliability is actually improving over time or just looking fine on a dashboard that only tracks the crude binary case. It's also the foundation any credible external SLA commitment eventually has to be built on, since SLA and accuracy credit terms you offer customers are meaningless if the underlying definition of "degraded" is left vague.
FAQ
What sample size is enough for the quality-degradation eval? Enough to detect a meaningful score shift reliably, typically starting in the low hundreds of sampled interactions per period and scaling with traffic; the exact number depends on your baseline variance, which you should measure before committing to a threshold.
Should quality degradation count against the same SLA as full outages? Not necessarily the same numeric target, but it should be tracked and reported with equal seriousness, since customer-perceived reliability is driven more by quality degradation than by rare full outages in most AI agent deployments.
Who should own this methodology internally? Whoever owns the post-launch ROI and metrics cadence is a natural owner, since reliability methodology and ongoing performance tracking are closely related disciplines.
How often should the thresholds be revisited? At minimum whenever the underlying model or architecture changes meaningfully, since a threshold calibrated against one model's failure modes won't necessarily transfer cleanly to another.