How to Decide Which AI Agent Conversations Get Routed to a Cheaper Model vs. a More Capable One

Quick answer

Model tiering routes each conversation turn to the cheapest model capable of handling it well, based on signals like query complexity, whether the request touches a high-stakes action (payments, account changes, health or legal topics), and the confidence of a lightweight classifier or the smaller model itself. The routing decision should default to the more capable model whenever the classifier is unsure, since the cost of escalating a simple query is a few cents, while the cost of under-routing a complex or high-stakes one is a bad customer experience or a compliance problem.

This is a different decision from picking one model for your agent

Our post on choosing a foundation model for an AI agent covers the one-time decision of which model or models to build against. Model tiering is a runtime decision made on every single conversation turn: given the models you've already chosen to support, which one should handle this specific request. It's the difference between choosing what's in your toolbox and deciding which tool to pick up for the job in front of you right now.

Why tiering exists at all

A capable, expensive model can handle every request correctly, but most production agent traffic is disproportionately simple: order status lookups, basic FAQ answers, straightforward confirmations. Running all of that through your most capable and most expensive model is paying premium-model pricing for requests a much cheaper model would answer identically. Our post on inference cost at scale covers just how much this adds up once volume gets large; tiering is the primary mechanism for controlling that cost without degrading quality on the requests that actually need the stronger model.

What determines the routing decision

Query complexity signals. Length, ambiguity, and the number of distinct sub-questions in a single message are decent cheap proxies for how much reasoning a request needs. A one-line factual question routes differently than a multi-part request requiring the agent to reconcile several pieces of information.

Stakes of the action involved. Anything touching a payment, account change, cancellation, or a regulated topic like health or legal information should route to the more capable model regardless of how simple the query looks on the surface, since the cost of a mistake in these categories is disproportionately higher than the savings from a cheaper model.

Confidence signals from the cheaper model itself. A common and effective pattern: let the cheap model attempt the response first, and if its own confidence score, or a lightweight secondary check, is low, escalate to the stronger model rather than returning a low-confidence answer. This is sometimes called a cascade architecture, and it captures much of the cost benefit of tiering without requiring a perfect upfront classifier.

Conversation history and prior escalation. If a conversation has already required correction or has shown early signs of customer frustration, covered in our post on the human-in-the-loop approval bottleneck, route subsequent turns to the more capable model even if the individual query looks simple in isolation, since the cost of another misstep in an already-strained conversation is higher than usual.

Building the routing layer

1. Start with a small, fast classifier, not a rules engine that tries to enumerate every case. A lightweight model or even a simple embedding-based classifier trained on your own historical conversation data usually outperforms hand-written routing rules and is far cheaper to maintain as your product surface changes.
2. Default to the stronger model on low confidence, always. The asymmetry in cost between over-routing and under-routing means ties should never go to the cheaper model.
3. Log every routing decision alongside the eventual outcome, so you can measure whether the cheap-model tier is actually producing acceptable quality, not just assuming the classifier is working because costs went down. This connects directly to your broader quality metrics program, covered in our post on ongoing production quality metrics for AI agents.
4. Revisit tier boundaries periodically, not just once. Model capabilities and pricing both shift over time; a routing threshold set when you launched can become stale, sending traffic to a tier that's no longer the right cost-quality tradeoff months later.

The failure mode to design against specifically

The most damaging tiering mistake isn't picking the wrong model for an average request, it's confidently and cheaply misrouting a high-stakes request that looked simple on the surface. A one-line question about a medication interaction or a large account cancellation can look exactly as short and simple as a one-line question about store hours to a naive complexity classifier. This is why stakes-based routing rules should sit above and override complexity-based routing, not the other way around, regardless of how well the complexity classifier is performing on average.

FAQ

Does model tiering add noticeable latency from the routing decision itself? A well-built lightweight classifier adds negligible latency compared to the model call itself. Cascade architectures, where the cheap model attempts first, can add latency on the fraction of requests that get escalated, which is a tradeoff worth measuring against the cost savings for your specific traffic pattern.

How many tiers should a production system realistically use? Two or three is typical and usually sufficient: a fast, cheap default tier, a mid-tier for moderately complex requests, and a top-tier reserved for high-stakes or low-confidence cases. More tiers than that tend to add routing complexity without a proportional cost benefit.

Should routing decisions be visible to the customer? Generally no, this is an internal cost-optimization detail rather than something that needs surfacing, provided the quality bar is genuinely consistent across tiers for the request types routed to each one. If tiering visibly degrades quality on certain request types, that's a signal to fix the routing thresholds, not to disclose the tiering itself.

Can tiering be applied retroactively to an agent that currently uses a single model for everything? Yes, and it's often the highest-leverage cost optimization available for an agent that's already stable in production, since it requires no change to the underlying prompts or capabilities, only a routing layer added in front of the existing single-model setup.