Quick answerA customer complaint about the AI agent should trigger a formal internal incident review, rather than a routine one-off correction, when at least one of three conditions is met: the same underlying cause is plausibly affecting other customers, not just the one who complained; the complaint involves a decision category with real financial, legal, or safety weight regardless of how many customers it touched; or a similar complaint has already been logged and corrected individually within a defined recent window, indicating a pattern rather than an isolated miss. Define these triggers explicitly rather than leaving the incident-or-not call to whoever happens to be handling the complaint that day.
The decision this covers sits upstream of both existing processes
Designing a customer-facing flag-this-answer-as-wrong mechanism covers how a customer reports a problem. What a company should communicate to customers after an AI agent has a production incident covers how to talk about an incident once it has already been identified as one. Neither answers the question in between: when does an individual complaint stop being a one-off correction and start being an incident that needs its own internal review, root-cause process, and possibly broader customer notification? Without an explicit threshold, that call defaults to whoever is triaging the complaint, and their bar for "this seems bad enough to escalate" will vary by person and by day.
Why scope, not severity, is the first thing to check
A single customer getting one wrong answer, corrected on the spot, is normal operation for any AI agent at scale and does not need a formal review every time. What changes the calculus is scope: could this same failure be affecting other customers right now without anyone else having complained yet. Any complaint tied to a rule, a knowledge-base entry, or a decision path that applies broadly, not just to that one customer's specific situation, should default to a quick scope check (has this pattern shown up elsewhere in recent logs) before deciding whether it is isolated or systemic.
The categories that skip the volume threshold entirely
Some complaint categories deserve a formal review regardless of how many customers were affected, because the potential harm per incident is high even at a sample size of one: anything touching a financial transaction the agent got wrong, a safety-relevant response, or a legal or compliance boundary the agent crossed. Waiting for a second complaint before treating these as an incident optimizes for the wrong thing, treating volume as the trigger when severity alone should be sufficient.
Using repeat-complaint patterns as a backstop, not the primary signal
What changes once a customer is a confirmed repeat complainant covers how the company's response to one customer's escalating pattern should change over time. The threshold described here is different and complementary: tracking whether the same type of complaint, from different customers, is recurring within a defined window, say two or more independent reports of the same underlying issue within 30 days. That pattern should trigger a formal review even if any single instance looked minor and got corrected individually, since individually-minor, collectively-recurring is exactly the shape of problem a case-by-case correction process is built to miss.
FAQ
Who should own deciding whether a complaint crosses the incident threshold? Define named criteria in advance so the initial triage person can apply them directly, with a clear escalation path to whoever owns AI incident response for borderline cases rather than requiring every complaint to go through a committee.
Does a false positive, escalating something that turns out to be isolated, carry real cost? Some cost, mostly review time, but it is a much cheaper mistake than the reverse: under-escalating a systemic issue that keeps quietly affecting customers because no single complaint looked severe enough on its own.
Should this threshold be documented publicly, or is it purely an internal process? Keep the specific triggers internal, since publishing exact thresholds can invite testing the boundary, but be transparent externally about the fact that a formal review process exists and what customers can expect if their complaint triggers one.