Quick answerA recurring AI agent transparency report should cover a consistent set of metrics over a fixed period: overall accuracy on scoped categories, containment and escalation rates, categories of known limitation, and a summary of significant incidents already disclosed individually. Publish it on a fixed cadence, quarterly is common, and use the same metric definitions every period so readers can actually track trend rather than a fresh set of numbers each time.
Distinct from disclosure rules and from single-incident communication
Whether you are legally or ethically obligated to tell a customer they are talking to an AI at all is a separate question, covered in do you have to tell customers they're talking to an AI agent. That is a one-time, per-conversation disclosure decision. Communicating about one specific incident after it happens is also a separate practice, covered in should your AI agent proactively tell a customer about its own mistake. That is reactive and incident-specific.
A recurring transparency report is neither of those. It is a standing, periodic publication that aggregates performance over a defined window, independent of whether any single notable incident occurred during that window.
Core metrics to include every period
Keep the metric set stable across reporting periods rather than changing what you measure each time, since the value of a recurring report comes from comparability over time. A reasonable baseline set: overall accuracy rate on the same scoped, tool-sourced categories most companies already use for internal customer commitments, containment rate versus human escalation rate, median and worst-case response time, and a plain-language summary of the categories of question the agent still handles poorly.
This is a narrower, more external-facing set than what goes in an internal board dashboard; the full internal view is covered in what AI agent metrics belong in front of the board or CEO. A public or customer-facing report should include only what is meaningful and defensible to a non-technical outside reader, not the full internal operating dashboard.
From the team
We build production AI systems for startups.
LLM pipelines, RAG, and agent workflows that hold up under real traffic — not just in the demo.
Referencing individually disclosed incidents without re-litigating them
If a significant incident was already communicated on its own, per the proactive disclosure practice above, the recurring report should reference it briefly rather than omit it or repeat the full original disclosure. A short summary line with a link to the original communication keeps the report honest without turning it into a rehash of every past incident.
Cadence and where to publish it
Quarterly is a common cadence, frequent enough to stay current, infrequent enough to be worth the preparation effort. Publish it somewhere durable and linkable, a dedicated page rather than a one-off blog post, so each new period's report can reference the trend against prior periods in the same place.
FAQ
Is a transparency report only for large companies?
No, though the format can scale down. Even a short quarterly summary builds more trust than no reporting at all, and it is easier to start small and expand than to retrofit credibility later.
Should the report include failures the company caught internally but no customer ever noticed?
Aggregate metrics like overall accuracy rate should include them, since excluding internally-caught errors would understate the true rate. Individual incident call-outs are a separate, narrower category.
What if the numbers look bad in a given quarter?
Publish them anyway, with context. A transparency report that only appears in good quarters is not a transparency report, and the credibility cost of an obvious gap is worse than one weak quarter honestly reported.