Quick answerWhat if a customer specifically asks whether they are in a canary group? Tell them honestly. Refusing to answer a direct question about a canary group a customer has already noticed and asked about turns a normal testing practice into something that looks like active concealment. Should canary membership itself ever be sold or marketed as an early-access perk? That is a different, opt-in program (an early-access or beta tier), not a canary rollout in the sense this post covers, since a canary test's whole value comes from the cohort behaving like an ordinary, unaware customer population.
Quick answer
A customer in a canary group is, by definition, getting materially different AI agent behavior than everyone else, and they did not opt into that difference. You do not owe every customer advance notice of every canary test, since that would defeat the point of testing quietly, but you do owe three things once the difference is material: an easy, working path to reach the standard behavior if the canary version causes them a problem, a support process that recognizes canary membership instead of treating their experience as a bug report against the general-availability version, and a clear internal threshold for when a canary difference becomes large enough that it must be disclosed proactively rather than only on request.
This is a different question than how you design the canary test
How to canary-test an AI agent in production before full rollout covers the internal engineering question: cohort sizing, rollback triggers, statistical confidence before graduating a change to general availability. None of that internal methodology tells you what you owe the customer who is living inside the test right now. A canary design can be methodologically perfect and still leave real customers with a materially worse or just different experience with no idea why, and no path back to the standard version if they want one.
Why silent canary testing is normally fine, and when it stops being fine
Small canary differences (a slightly different phrasing, a marginally different response time, an internal ranking tweak) do not need individual disclosure; testing this quietly is exactly the point, and disclosing every micro-experiment would make the AI agent's behavior harder for customers to understand, not easier. The threshold flips once the canary difference is large enough that a customer, if they knew, would reasonably want to act differently: a canary version that declines a request the general-availability version would approve, one that takes meaningfully longer, or one that behaves in a way indistinguishable from a bug from the customer's seat. At that threshold, silence starts to look like withholding a material fact rather than ordinary product testing.
Give support a canary-aware escape hatch
The single highest-value control is operational, not disclosure-based: make sure a support agent, human or AI, can recognize that a specific customer's odd experience traces back to canary membership, and can move that customer to the general-availability behavior on request without a multi-day investigation. Without this, a customer's confusing experience gets treated as an isolated one-off bug report, the canary team never learns the pattern is systemic, and the customer is left stuck in a worse experience with no visible way out.
Tie this to your standing behavior-change communication practice
Once a canary graduates and actually ships to everyone, it becomes a real behavior change and should flow into your quarterly proactive behavior-change update cadence like any other shipped change, not be treated as invisible just because it started life as a quiet experiment.
This sits on top of your baseline AI-disclosure obligation, not instead of it
Whether you have to tell customers they are talking to an AI agent at all is a separate, baseline question every customer-facing agent has to answer regardless of canary status. A customer already told they are talking to an AI agent still has not been told which behavioral variant of that agent they are getting; the canary disclosure question covered here is a layer on top of the baseline one, not a substitute for it.