The Symptom

Two complaints, same root. Users said the product "lagged" — multi-second waits on actions that felt like they should be instant. And the finance side flagged that the AI infrastructure bill was outgrowing user growth, which meant unit economics were quietly getting worse the more the product succeeded. Both were pointed at the model. Neither was the model's fault.

What We Found

The expensive, slow work was in the plumbing:

• Recomputing the unchanged. The system re-embedded and re-fetched content on every request, including documents that hadn't changed since the last time. A large share of the compute was spent re-deriving things that were already known.
• Oversized context. Each call stuffed in far more retrieved context than the answer needed — paying, in latency and tokens, for material the model then ignored.
• Sequential everything. Independent steps ran one after another instead of in parallel, so total latency was the sum of the steps rather than the slowest one.
• One big model for every step. Trivial sub-steps went to the same heavyweight model as the hard ones, paying premium latency and cost for work a smaller model could do instantly.

What We Changed

• A caching layer (Redis) for embeddings and frequent retrievals, so unchanged work is computed once, not every request.
• Stopped re-embedding unchanged content — embeddings are refreshed only when the underlying content actually changes.
• Trimmed and ranked context down to what the answer needs, cutting both tokens and time.
• Parallelised independent steps, so latency tracks the slowest step instead of the sum.
• Routed simple steps to lighter models, reserving the heavyweight model for the work that needs it.

The Result

Response time dropped by roughly 25%, and the changes that produced it — caching, trimming, routing — also cut the volume of paid model calls, so the cost curve bent down in step with the latency. The product stopped getting proportionally more expensive as it grew, which was the outcome that actually mattered to the business.

The Lesson

The latency and cost of an AI product usually live in the retrieval and orchestration path, not in the model itself. Before you upgrade the model (more cost) or downgrade it (worse output), instrument where the time and the tokens actually go. Most teams discover they're paying, over and over, to recompute things that never changed.

When This Doesn't Apply

At low volume, this is premature optimisation. Caching, routing, and parallelism add moving parts, and if you're pre-traffic that complexity buys you nothing — ship first, optimise when the load is real. The payoff arrives when request volume and the bill are both large enough that recomputed work is actually costing you.

FAQ

Wasn't the model the reason it was slow and expensive?
Almost never on its own. The model call is one step; the retrieval and orchestration around it are where redundant compute hides. We changed those and left the model alone.

What is "re-embedding waste"?
Regenerating vector embeddings for content that hasn't changed since last time. It's pure recomputation — you pay full price to re-derive a result you already had.

Does caching make answers stale?
Only if you cache the wrong things. Embeddings of unchanged content are safe to cache; you invalidate on content change. Volatile results stay uncached.

How do you decide what to route to a smaller model?
By step. Classification, routing, and simple extraction go to a light model; synthesis and reasoning stay on the heavy one. You match model cost to step difficulty.