Your Model Provider Changed How the Model Behaves and Never Told You: What to Do About Silent Drift

Quick answer

Silent drift happens when a provider changes a model's behavior behind the same API endpoint and version string you have always called, with no deprecation notice and no changelog entry, so your first signal is your own eval scores or customer complaints moving without any code change on your side. Run your golden eval set against the live production endpoint on a fixed schedule, not just before a release, so a drift shows up as a score change you can point to a date. When it happens, do not assume the provider will confirm it; document the before-and-after behavior yourself, route customer-facing consequences through your incident process immediately, and treat the vendor conversation as evidence-gathering rather than an expectation of a quick admission.

Why this is a different risk than a planned migration or a forced deprecation

A planned migration is a decision you make on your own timeline, covered in what should actually trigger an AI agent provider migration. A forced deprecation is at least announced, even if the timeline is inconvenient, covered in your model provider just announced a deprecation date, now what. Silent drift has neither a decision point nor an announcement. The same model name, the same API version, the same documented capabilities, but the actual output distribution has shifted, often because the provider updated weights, safety tuning, or a routing layer behind the scenes without treating it as a versioned release.

Detect it with continuous evaluation, not just pre-release testing

Most teams only run their eval suite before shipping a change on their own side, which means a provider-side drift can go undetected for weeks because nothing on your end triggered a test run. Schedule your existing golden eval set, built the same way described in building a golden evaluation dataset for your AI agent, to run against the live production endpoint on a recurring cadence regardless of whether you have shipped anything. A meaningful score movement with no corresponding change in your own prompt, code, or data is the clearest signal of drift you will get, and it gives you a specific date range to point to when you raise it with the provider.

Do not expect the provider to confirm it happened

Providers are often reluctant to confirm an undocumented behavior change, partly because "the model behaves differently" is hard to prove from their side without your specific evals, and partly because acknowledging silent changes undermines the version-stability promise most providers make publicly. Treat your own eval logs, sampled transcripts, and timestamps as the record of what happened, independent of whether the provider ever agrees. This is the same discipline as documenting AI agent reasonableness for disputes, applied to a vendor relationship instead of a customer one.

Put a contractual tripwire in place before it happens again

If your current vendor contract only obligates the provider to notify you of deprecations, it says nothing about silent behavioral changes within the same version, which is precisely the gap that makes this risk possible. When negotiating or renewing a contract, ask for a defined behavioral-stability commitment tied to a named eval or benchmark, not just an API version number, alongside the other terms discussed in negotiating SLAs and support terms in an AI vendor contract. A version number guarantees the interface; only a behavioral commitment guarantees the outputs.

Have a rollback path even when nothing changed on your end

Because silent drift is not tied to a deployment you control, you cannot simply revert your own last change to fix it. Maintain the ability to pin to a previous model snapshot or route a percentage of traffic to a fallback provider, the same infrastructure described in AI agent failover when a model provider has an outage, so a confirmed drift gives you an immediate mitigation option rather than a multi-week wait for the provider to fix or revert their own change.

FAQ

How is silent drift different from normal model variance? Normal sampling variance is bounded and stable over time for the same inputs. Drift shows up as a sustained shift in eval scores or output patterns that persists across many runs and does not revert on retry, which is what separates it from ordinary non-determinism.

Can you detect silent drift without a golden eval set? It is much harder. Customer complaints and support ticket volume are lagging, noisy signals. A fixed, versioned eval set run on a schedule is the fastest reliable detector because it isolates the model as the only variable.

Should you name the provider publicly if this happens? Usually not as a first move. Raise it privately with evidence first; escalate publicly only if the provider denies a well-documented, reproducible shift and customer impact is ongoing.

Does switching to open-weight models eliminate this risk? It reduces it for the weights themselves, since you control when they change, but does not eliminate it if you rely on a hosted inference provider that can still change routing, quantization, or system-level settings without notice.