Quick answerRunning two behavior variants of your own agent side by side on live traffic is a legitimate way to validate a change before a full rollout, but it puts real customers into an experiment they did not opt into, which changes what you owe them. At minimum, keep both variants inside your existing disclosure and quality bar, meaning neither version is allowed to be worse than your published standard even temporarily, cap the size and duration of the test, and have a clear, fast rollback trigger if one variant's error rate or complaint rate diverges. Do not treat this as equivalent to a marketing landing-page test; a wrong answer in a customer service conversation has consequences a button-color test does not. ---
How this differs from testing the agent against your old process
Our post on how to A/B test an AI agent against your existing process before a full rollout covers the first-launch question: does the agent perform at least as well as what it is replacing. That test compares a known process against an unknown one. This test compares two unknowns against each other, version A and version B of the same agent, and both are live in production simultaneously, which means a bad variant cannot be caught by comparing against a stable baseline the way the launch test can.
The consent question you cannot fully solve, only manage
Customers did not sign up to be randomly assigned into a behavioral experiment, and there is no practical way to get informed consent for a specific conversation-level variant assignment without breaking the experiment itself. What you can do is stay inside the disclosure commitments already covered in whether you have to tell customers they're talking to an AI agent, meaning customers still know they are talking to an AI, even if they do not know which internal variant, and make sure neither variant is allowed to fall below the quality bar you have already committed to publicly. Treat the test as an internal engineering decision with a real customer-facing floor, not a free pass to ship an unvetted variant.
Version control has to be tight enough to explain what a customer experienced
If a customer later disputes what the agent told them, you need to be able to reconstruct exactly which variant they were on and what that variant's prompt and logic actually did at that timestamp. This depends on the same discipline covered in how to version and roll back AI agent prompts safely in production: every variant needs its own tracked version, tied to the specific conversations it served, not a loose "we were testing something" note in a deploy log.
Set the rollback trigger before you start, not after you see results
Decide in advance what divergence between the two variants ends the test immediately, an error-rate gap, a complaint-rate gap, or a specific category of wrong answer, rather than watching the numbers drift and deciding case by case. A trigger defined after you already see a bad signal invites rationalizing it away.
FAQ
Is a 50/50 split always the right size for this kind of test?
No. Start smaller, often 10 to 20 percent on the new variant, specifically because both variants are live and unconsented; a smaller slice limits how many customers are affected if the new variant underperforms.
Do we need to tell customers afterward which variant they were in?
Most companies do not, as long as both variants stayed inside the existing disclosure and quality commitments; the obligation is to keep the floor high during the test, not to retroactively debrief every participant.