Quick answerWatch for the specific pattern of customers who set the standing human-only preference right before an action the AI agent would normally screen more carefully, a refund request, a fraud-sensitive change, or a discount ask, rather than the general pattern of frequent human contact the preference is meant to serve honestly. Keep the preference itself easy to set, since making it hard defeats the point, but route flagged accounts to a human who still has the agent's normal screening checks available as a checklist, not to a human working from a blank slate who assumes the customer already passed whatever the agent would have checked. Treat repeated short-lived toggling of the preference, set right before a sensitive request and turned off right after, as its own signal worth reviewing. ---
This is the abuse question, not the design question
Our post on whether an always-route-me-to-a-human preference should persist across sessions covers how to store the preference, what language should count as setting it, and how to give customers visibility and an easy way to turn it off. That post assumes good-faith use. This post covers the minority of cases where the same standing preference becomes a deliberate way to route around the AI agent's normal checks entirely, using the human channel as a way to reach someone who might approve something the agent was specifically built to catch.
Why humans are not automatically a safer target for this kind of gaming
It is tempting to assume a human reviewer is always a stronger check than an AI agent, but a human without the agent's structured screening checklist in front of them, working from a fresh conversation with no flags raised, can actually be an easier target for a customer who knows exactly what they are trying to get past. The fix is not to weaken the human channel's authority, it is to make sure the human still sees whatever risk signals the agent would have surfaced, the same context-completeness principle covered in what context a human should see the instant an AI agent escalates, applied here specifically to accounts with the standing preference active.
The pattern to actually watch for
A single instance of setting the preference right before a refund request is not evidence of anything; plenty of customers genuinely prefer a human for exactly that kind of request. The signal worth building detection for is repetition: the same account toggling the preference on shortly before a sensitive action and off again shortly after, across multiple occurrences, which looks much more like probing for the easiest path than like a stable communication preference.
Borrow the abuse-detection posture from trial and freemium design
The general discipline here is similar to what is covered in designing trial and freemium AI agent access without getting abused: keep the legitimate path completely unrestricted for the vast majority of honest users, and build detection specifically around the narrow behavioral pattern that distinguishes abuse from normal use, rather than adding friction that punishes everyone to stop a small minority.
FAQ
Should setting the preference ever be denied or reversed by the company?
Only in confirmed abuse cases with a documented pattern, never on a single instance, since reversing a customer's own stated preference without strong evidence damages trust with the much larger group of honest users.
Does this abuse pattern justify adding friction to setting the preference in the first place?
No. Adding friction to the honest majority to stop a small pattern of misuse is exactly the tradeoff worth avoiding; detect the pattern after the fact instead of gating the preference upfront.