Quick answerBuild a specific policy that treats a customer who may be a scam victim as someone to protect, not someone to process. The agent should recognize known scam patterns relevant to your platform, such as a buyer being pressured to pay outside your protected checkout or a seller being asked to ship before payment clears, pause whatever transaction is in motion, and clearly warn the customer in plain language before offering to connect them with a human trust and safety reviewer. This is a different policy than fraud detection aimed at protecting the company itself; here the company's own platform is just the venue, and the customer in front of the agent is the one at risk.
This is a different problem than fraud against the company
Most existing fraud tooling is built to protect the company: detecting stolen payment methods, fake accounts, or abuse of a promotion. A customer being scammed by another party through your platform is a distinct scenario, where your company's exposure is reputational and duty-of-care rather than direct financial loss, and the harm lands on the customer, not on you. This deserves its own detection logic and its own response script, rather than being bolted onto an existing anti-fraud system tuned for a different threat model entirely. The agent's job here is closer to a warning system than a security control.
Recognize the pattern, then pause before it completes
Effective detection depends on recognizing platform-specific patterns rather than generic fraud signals: a buyer being steered to pay outside your protected checkout flow, a seller being pressured to ship before payment clears, or a conversation that shows classic urgency and secrecy pressure tactics. Once the agent recognizes one of these patterns, the highest-value action is pausing the transaction in motion and surfacing a plain-language warning before the customer commits, rather than logging the concern silently for later review. This is where the agent's sentiment and escalation handling for upset customers becomes directly relevant, since a customer being warned mid-scam is often confused, defensive, or in a hurry, and the tone of the warning needs to account for that reaction rather than assuming a calm, receptive audience.
From the team
We build production AI systems for startups.
LLM pipelines, RAG, and agent workflows that hold up under real traffic — not just in the demo.
Route to a human reviewer, and keep the record
Any suspected scam pattern involving money movement should route to a human trust and safety reviewer quickly, particularly where the agent has any ability to see into financial system access on the company's side, since a scam in progress can sometimes be stopped outright if a payment or transfer has not yet finalized. Keep a clear record of what the agent detected, what it warned the customer about, and what happened next, using the same discipline behind documenting AI agent reasonableness for a future dispute. That record matters twice over here: it protects the company if a customer later claims the platform ignored clear warning signs, and it gives the trust and safety team a growing library of real scam patterns to tune future detection against.
FAQ
What if the agent's suspicion turns out to be wrong and the transaction was legitimate? Design the warning to be low-friction and non-accusatory, framed as a caution rather than an accusation, so a false positive costs the customer a few extra seconds rather than an uncomfortable confrontation. A warning that is easy to dismiss when wrong is better than one so aggressive it damages a legitimate transaction.
Should the agent ever block a transaction outright rather than just warn? Only where the platform's own policy already allows a hard block for a defined pattern, and even then it should be a narrow, well-tested list. Most cases are better served by a clear warning plus an easy path to a human, since an outright block on a false positive creates its own support burden and trust cost.
Does this policy differ for a marketplace with both buyers and sellers? Yes, the specific scam patterns differ for each side, and the agent needs pattern recognition tuned separately for buyer-side risks like fake listings and seller-side risks like non-paying buyers pressuring early shipment, rather than a single generic scam detector applied to both roles.