Quick answerTrigger a full independent re-audit of decision-making quality, not just a routine eval run, whenever an underlying model version change meets any of these conditions: the change is a major version jump rather than a minor patch, the provider's own release notes flag changes to reasoning, alignment, or safety behavior, or your own spot-check evals show a shift, in either direction, on a protected or high-stakes decision category such as pricing, eligibility, or content moderation. A model swap can quietly change which customers get approved, denied, or treated differently in ways your existing eval suite, built around the prior model's failure modes, was never designed to catch.
A different concern than security or general audit cadence
When a full security re-audit is needed already lists a foundation model upgrade as one of its four triggers, but that post's re-audit is about security posture: new tool or data access, adversarial probing resistance, and attack surface. It does not evaluate whether the model's actual decisions on real customer cases have shifted in fairness or accuracy. The general independent-audit-versus-internal-evals trigger post covers broader conditions, entering a regulated vertical, decisions affecting people at scale, complaint patterns, that can apply regardless of whether a model change happened at all. This post is specifically about the decision-quality risk introduced by the model swap itself, which neither of those fully covers.
It is also distinct from how often a regulated industry should refresh its knowledge base for compliance, which is a content-freshness question about the information the agent has access to, not a question about how the model itself reasons over that information.
Why routine monitoring alone misses this
Standard production monitoring is built to catch drift and degradation against a baseline established under the previous model version. A version swap changes the baseline itself, which means monitoring can show every metric looking stable while the actual distribution of decisions underneath has shifted, because the new model reaches similar aggregate outcomes through a different reasoning path with different edge-case behavior. A model that is measurably more accurate on average can still be meaningfully worse on a specific protected category or edge case that your existing dashboards were not built to isolate, and that only a targeted re-audit, run specifically to compare pre- and post-swap decisions on the same held-out case set, will surface.
The trigger conditions in detail
Major version jump, not a minor patch. Providers typically signal the scope of a change through their own versioning. A minor patch focused on latency or cost rarely warrants a full re-audit. A major version change, especially one the provider describes as a substantial architecture or training update, should trigger one by default, before waiting for any downstream signal that something went wrong.
Provider release notes flagging reasoning, alignment, or safety changes. Read release notes specifically for language about changed refusal behavior, updated safety training, or shifts in how the model handles ambiguous or high-stakes requests. These are direct signals that decision behavior, not just capability, has changed.
Your own spot-check evals show any shift on high-stakes categories. Before fully rolling out a new model version, run your existing eval suite against a fixed set of past real cases in categories like pricing decisions, eligibility determinations, or content moderation calls, and compare outputs directly against what the prior model produced on the same cases. Any meaningful divergence, better or worse, on these specific categories is grounds for a full re-audit before full rollout, not just a note in a release changelog.
Building this into the rollout process itself
The strongest version of this trigger is proactive: build the comparison eval run into the model-upgrade rollout process as a required gate, rather than treating re-audit as something triggered reactively after a complaint pattern emerges. By the time a complaint pattern is visible externally, the new model version has typically already been serving live traffic long enough to have made a meaningful number of the decisions the re-audit exists to catch.
FAQ
Does a provider silently changing model behavior without a version bump count as a trigger? Yes, and this is arguably the harder case to catch, since there is no explicit version change to key off. Ongoing spot-check evals run on a fixed cadence, independent of any known provider change, are the main defense against this scenario, sometimes called silent model drift.
Who should run the re-audit, internal team or a third party? For most version changes, an internal team running the same rigorous comparison methodology is sufficient. Escalate to a third-party independent auditor when the conditions from the general audit-trigger framework also apply, such as a regulated vertical or decisions affecting people's finances or legal standing at scale, since that is a separate, higher bar than the model-change trigger alone.
How long should the pre- and post-swap comparison eval set be maintained? Keep it as a permanent, versioned asset that gets appended to over time rather than rebuilt from scratch for each model change, so each new comparison benefits from the accumulated edge cases identified in prior version transitions.