When "Slower and Worse" Should Trigger the Same Response as "Down"

Quick answer

Set explicit, pre-agreed thresholds on the metrics that matter most, latency, accuracy or containment rate, and error rate, and trigger the same incident response process for a sustained threshold breach as you would for a full outage, just with a different initial severity label. The mistake most teams make is only building alerting for binary availability (is the API responding at all), which means a provider that is technically up but running 3x slower or noticeably less accurate can degrade customer experience for hours or days before anyone formally notices, simply because nothing ever went fully down.

Full outage and partial degradation need different detection, same response discipline

Handling a complete model provider outage, where the API stops responding entirely, is a relatively well understood problem covered in AI agent failover when a model provider has an outage: detect the failure, fail over to a backup provider or degraded mode, notify stakeholders, recover when the primary comes back. The detection step there is comparatively easy, because a fully down API produces an unambiguous signal.

Partial degradation is harder precisely because the signal is ambiguous. A provider running slower than usual, returning slightly less accurate responses, or timing out on a subset of requests looks, from a pure uptime dashboard, like the system is fine. Without deliberately built thresholds for degraded-but-technically-working states, this failure mode can persist far longer than a full outage would, because nothing forces anyone to notice it.

Pick the right metrics and set thresholds before you need them

Three metrics typically matter most for detecting meaningful degradation, each needs its own threshold, decided in advance, not improvised during an incident:

• Latency. Track p50 and p95 response time against a rolling baseline, not an absolute number, since acceptable latency varies by use case. A sustained doubling of p95 latency over a meaningful window (not a single spike) is a reasonable default trigger for at least an initial-severity incident.
• Accuracy or containment rate. This is the hardest to detect in real time, since you often do not have ground truth for a live conversation. Proxy signals work reasonably well: a sudden rise in escalation rate, a sudden rise in negative sentiment, or a drop in the containment rate covered in measuring and improving AI agent containment rate, sustained over enough volume to rule out normal noise.
• Error and timeout rate. Even without a full outage, an elevated but non-zero error rate (say, 5 to 15 percent of requests failing or timing out) deserves its own threshold distinct from the "fully down" threshold used for full failover triggers.

Treat threshold breaches as graduated severity, not a single alarm

Build at least two tiers so that degraded performance does not get ignored, but also does not trigger the exact same all-hands response as a full outage every time:

1. Watch tier. A metric crosses its threshold but has not sustained long enough, or is not severe enough, to warrant customer-facing action yet. This triggers internal monitoring attention and a named owner, but not a public status update.
2. Incident tier. The threshold breach is sustained past a defined duration, or crosses a second, more severe threshold. This triggers the same incident response process used for a full outage, following the pattern covered in what to communicate to customers after a production incident, including the same postmortem discipline, even though the underlying cause was degradation rather than a hard failure.

Watch for degradation that never crosses a hard threshold but still matters

Some degradation is gradual enough that it never triggers a discrete threshold breach, it just slowly gets worse over weeks. This is where the production quality metrics discussed in what production quality metrics to track for a live AI agent earn their keep as a standing practice, not just an incident-response input: a slow trend line, reviewed on a regular cadence rather than only checked reactively, catches this kind of degradation before it ever becomes acute enough to trip an alert.

FAQ

How is this different from normal performance monitoring? Normal monitoring tracks metrics continuously. This is specifically about defining, in advance, the point at which a metric's decline should escalate to a formal incident response process, rather than staying invisible on a dashboard nobody is actively watching.

Should degraded performance always trigger the same severity as a full outage? Not automatically. A graduated tier system, watch versus incident, lets you respond appropriately without either ignoring real degradation or treating every minor fluctuation as a crisis.

What is the hardest metric to threshold correctly? Accuracy or quality, since it is rarely directly measurable in real time. Most teams rely on proxy signals like escalation rate or sentiment shift, sustained over enough volume to distinguish real degradation from normal noise.

Do customers need to be notified about a watch-tier degradation? Generally no, watch-tier issues are for internal awareness. Customer notification is typically reserved for incident-tier breaches, following the same communication discipline used for full outages.