AI Center of Excellence Operating Model: What to Build Once vs. Per Business Unit

Quick answer

Split the decision along a single axis: build once, centrally, anything that is expensive to get right, low in business-unit-specific nuance, and risky if done inconsistently (model access and governance, evaluation tooling, guardrail and compliance layers, observability). Let each business unit build independently anything that depends on unit-specific domain knowledge, workflows, or customer relationships that a central team cannot reasonably know as well as the unit itself (prompts and use-case logic, unit-specific integrations, domain-tuned retrieval content). The operating model that fails is the one that tries to centralize everything, which creates a bottleneck the center of excellence cannot staff fast enough, or the one that centralizes nothing, which produces N different, inconsistent, redundantly-built governance and eval layers across N business units.

The core tension multi-unit companies run into

Once more than one business unit is building or running AI agents, a predictable tension shows up: business units want speed and control over their own use cases, while whoever is accountable for AI risk and consistency across the company wants shared standards and wants to avoid every unit reinventing infrastructure that could have been built once. Resolving this well is less about org-chart politics and more about correctly classifying which parts of the stack actually benefit from centralization and which parts genuinely need to live close to the business.

This question is related to, but distinct from, two other decisions companies often conflate with it. Whether to measure a specific internal platform team's own performance is a separate question from what that team's mandate should even be, covered in what KPIs belong in an internal AI platform team SLA. Whether and when to stand up a governance committee at all is also separate, covered in when does a company actually need an AI governance committee. This post assumes both questions are already answered, a platform team exists and is measured, a governance function exists, and asks the operating question underneath both: what should that platform team actually own centrally, on an ongoing basis, versus leave to each business unit.

What almost always belongs centralized

A few categories are close to universal candidates for centralization, because building them per-unit is expensive to duplicate and inconsistent execution creates real risk:

• Model access and vendor relationships. Negotiating model provider contracts, managing API keys and cost allocation, and handling provider migrations once, centrally, avoids N units each negotiating separately and N different vendor lock-in situations.
• Guardrail and compliance infrastructure. Prompt injection defenses, PII redaction, content policy enforcement, and audit logging are exactly the kind of thing where inconsistent per-unit implementation creates real exposure, since a weak guardrail in one unit's agent is a company-wide reputational and legal risk, not just that unit's problem.
• Evaluation tooling and golden datasets infrastructure. The tooling to run evals, the platform to track eval results over time, and the general methodology should be shared, even though the actual eval content, the specific test cases, needs to be unit-specific.
• Observability and production quality metrics infrastructure. The pipeline that captures conversation logs, tracks quality metrics, and supports incident triage (see production quality metrics you should be tracking) is expensive to build well and nearly identical in shape across units; building it once and letting units plug in their own dashboards on top is far cheaper than N separate observability stacks.

What almost always belongs with the business unit

The mirror-image list is just as important, since over-centralizing these creates a bottleneck that slows every unit down waiting on a central team that cannot possibly have deep expertise in every unit's domain:

• Prompt and use-case logic. The specific way an agent should handle a customer conversation in, say, a healthcare unit versus a logistics unit reflects domain knowledge the central team is unlikely to hold as deeply as the unit itself.
• Unit-specific system integrations. Connecting the agent to a unit's specific CRM, ERP, or legacy system is inherently unit-specific work; a central team taking this on becomes a queue every unit has to wait behind.
• Domain-tuned retrieval content and knowledge bases. The actual content an agent retrieves from, product catalogs, policy documents, domain FAQs, is owned and maintained by whoever is closest to that content being correct and current.

The failure modes on each side

Over-centralizing looks like: every business unit submitting a ticket to a single platform team to make even small prompt changes, a backlog that grows faster than the team can clear it, and units eventually building shadow, unsanctioned workarounds outside the sanctioned platform just to move at the speed their business needs, which is exactly the risk described in what a shadow AI problem looks like and how to govern it.

Under-centralizing looks like: five business units independently building their own PII redaction logic, of varying quality, five separate incident response processes for AI agent failures, and no shared way to compare eval results or quality metrics across units, so leadership cannot get a consistent picture of AI risk or performance company-wide. This is often what a governance committee gets stood up to fix reactively, after the fact, rather than what a well-designed operating model would have prevented from the start.

Making the split explicit and revisiting it

The specific line between "build once" and "build per-unit" should be written down, not left as an unstated norm, and it is worth revisiting as the company's AI usage matures. Early on, when only one or two units are using AI agents seriously, more can reasonably live per-unit since there is little duplication to save on. As more units adopt agents, the case for centralizing the shared-risk, shared-infrastructure layer strengthens, and the operating model should shift accordingly rather than staying fixed at whatever split made sense when only one unit was doing this work.

FAQ

How is this different from just deciding whether to have a platform team at all? This assumes a platform team or center of excellence already exists (or is being planned) and answers the operating question of what it should actually own centrally versus leave to individual business units, which is a different question from whether to have the function in the first place.

Should governance and operating-model decisions be made by the same committee? Not necessarily. A governance committee's job is often setting policy and risk tolerance; the operating-model split described here is more of an ongoing platform-management decision that may sit with a platform team lead reporting into that governance structure, rather than the committee itself.

What is the first sign a company has gotten this split wrong? Persistent unit-level frustration with turnaround time on prompt or integration changes is the clearest sign of over-centralization; inconsistent guardrails, duplicated infrastructure, or an incident in one unit that reveals no other unit has an equivalent safeguard is the clearest sign of under-centralization.

Does this operating model apply to companies with only one business unit? The build-once-vs-local split still applies conceptually between "platform-level" and "product-level" concerns even within a single unit, but the multi-unit reuse argument, avoiding duplicated infrastructure across units, is what makes this decision especially high-stakes for multi-BU companies specifically.