Quick answerAn AI agent's biggest fraud-prevention asset, that it evaluates every request the same way every time, is also the exact thing a repeat bad actor learns to exploit: once they find a phrasing or a sequence of claims that gets through, it works again on the next attempt with near-total reliability, and it works fast enough to run at a volume no human reviewer could sustain. Detect this by looking for consistency in the attacker's inputs across sessions and identities, not just anomalies in any single transaction, and slow the agent down deliberately, through friction or human routing, once a pattern match crosses a threshold, rather than relying on per-transaction fraud scoring alone.
A different threat model than transaction-level fraud
Setting a no-human-review refund dollar threshold is about balancing review cost against average fraud loss per transaction, and it is the right tool for the fraud that any refund process already has to deal with. It is not built to catch the pattern this post is about, which is not a single high-risk transaction but a low-value, high-repeat one: an attacker who finds a script that reliably produces a small refund and runs it fifty times across fifty identities, each one individually well under any reasonable review threshold.
This is also a different problem than recognizing a legitimate repeat-contact customer, which is about serving a real returning customer better. Here the goal is the opposite: recognizing that the same underlying actor is returning under different apparent identities, specifically to keep individual transactions below any single-identity detection threshold. And it is distinct from chargeback and card-network dispute handling, which governs a formal dispute process after a payment has already been contested, not the fraud-generation pattern itself.
Why speed and consistency create the exposure
A human reviewer processing a suspicious refund request brings inconsistent judgment across cases, which is a real weakness, but it is also an accidental defense: an attacker cannot easily predict which human will review their next attempt or exactly how that person will react to a slightly reworded claim. An AI agent removes that unpredictability. The same input produces close to the same output, every time, from every session. Once an attacker has reverse-engineered the input that gets a refund approved, they have effectively found a key that fits the lock indefinitely, and the agent's speed lets them turn that key as many times as they want before anyone notices a pattern.
What pattern-level detection looks like
- Fingerprint the claim, not just the identity. Track similarity across the actual language, order of information provided, and specific claim type used in refund requests, independent of the account or session making them. A cluster of near-identical claim scripts across nominally unrelated accounts is a stronger signal than any single account's history.
- Correlate low-level signals that individually look clean. Device fingerprints, payment instrument fragments, shipping address patterns, and request timing can all be individually unremarkable while collectively pointing at one actor running the same script repeatedly. Build this correlation as a background process, not something the agent itself has to reason about in real time.
- Add deliberate friction once a pattern match crosses a threshold, rather than waiting for a fraud team to manually flag it. This can be as simple as routing the next matching request to human review or requiring a verification step the automated script cannot pass, which breaks the economics of the attack without punishing the far larger population of legitimate customers making a similar-sounding request for unrelated reasons.
- Rotate or randomize non-substantive elements of the agent's own responses and verification prompts periodically. An attacker who has mapped exactly what triggers approval loses that map faster if the verification flow is not perfectly static, without changing the actual policy being applied.
FAQ
Doesn't rate-limiting by account already solve this? Rate-limiting a single account slows down an attacker using one identity, but this pattern is specifically designed to stay under any single-account threshold by spreading attempts across many accounts. Detection has to work at the claim-pattern level, across accounts, to catch it.
Should the agent be the one making the fraud-block decision? The agent should be the one surfacing the pattern match and applying the pre-approved friction response, such as routing to review. The decision to treat a new pattern as fraud in the first place should stay with a human fraud analyst who can weigh false-positive risk against a legitimate customer segment that happens to share surface-level similarities.
How often should the pattern-detection thresholds be reviewed? Review them at least quarterly, and immediately after any confirmed fraud incident, since attackers adjust their scripts in response to whatever friction was added last, and a threshold that worked six months ago can be stale against current attack patterns.