Quick answerA postmortem for a third-party silent model drift incident needs a section your standard template does not have: what you could and could not have detected before the incident, given that the provider gave no changelog entry or version bump. Write the root cause honestly as provider-caused, not as an internal process failure, but still own the detection gap, whether your golden eval set runs against the live production endpoint on a schedule or only before releases. Close with two separate action items, one for your own monitoring cadence and one for the vendor relationship, and route the vendor item through a contract or SLA conversation rather than an engineering fix, since there is no code change on your side to make.
The postmortem answers a different question than detecting the drift did
Detecting silent drift, covered in our post on what to do about foundation model silent drift, is about noticing your production agent is behaving differently for no announced reason. The postmortem starts after that: a customer-facing incident already happened, tickets came in or an SLA was missed, and now someone has to write up what occurred, why, and what changes going forward. The two are related but distinct disciplines, roughly the difference between a smoke detector and the fire department's incident report.
Resist the instinct to write root cause as your own process failure
Standard postmortem templates assume the root cause traces back to a decision your team made: a deploy, a config change, a missed test. When the actual root cause is the provider changing model behavior behind an unchanged API endpoint and version string, forcing that into the usual format produces a dishonest document, either blaming an internal team for something they could not have prevented, or silently omitting the provider's role to make the write-up fit the template. Name the provider and the specific behavior change explicitly in the root cause section, and separate it clearly from a hallucination or product bug triage finding, which is about classifying a single bad answer, not accounting for a systemic incident after the fact.
The one section every third-party-drift postmortem needs that a normal one does not
Add a detection-gap section that answers one question directly: at what point could your own monitoring have caught this before customers did, if at all. If your golden eval set only runs before releases, and no release happened, say so plainly rather than implying an eval gap that did not exist. This section is what separates a postmortem that produces a real action item from one that just documents what happened. It also gives you defensible language for the customer-facing side: you can say what you detected and when, without overstating what was detectable given your actual monitoring cadence at the time.
Split action items into two tracks, monitoring and vendor
A monitoring-track action item is something engineering can execute unilaterally, typically tightening the eval-run schedule against the live endpoint. A vendor-track action item usually cannot be executed unilaterally at all; it depends on the provider's cooperation, and belongs in a contract renewal or SLA conversation rather than an engineering backlog. Keep these visibly separate in the document. A single combined action-item list makes it easy to mistake a vendor-dependent item for something your own team controls, and to quietly let it die when the provider does not respond, since nothing on the internal side is technically blocking it. The write-up itself benefits from the same discipline as documenting AI agent reasonableness over time: a specific, dated, before-and-after account outlasts a vague summary if the incident is ever revisited months later.
FAQ
Should the postmortem name the model provider explicitly?
Yes. Naming the provider and the specific behavior change in the root cause section is what makes the document honest; softening it to protect the vendor relationship produces a write-up that misleads your own team about what they could have prevented.
Is this the same as a normal vendor-outage postmortem?
No. An outage is binary and usually has a status page confirming it. Silent drift has no confirmation from the vendor at all, which is why the detection-gap section matters more here than in a standard outage write-up.
What if the provider never confirms the change happened?
Write the postmortem based on your own before-and-after evidence regardless. Treat vendor confirmation as a nice-to-have for the write-up, not a prerequisite for closing it.