How to Migrate from a Legacy Rules-Based Chatbot to an LLM-Based Agent

Quick answer

Don't do a big-bang replacement. Run the LLM agent in shadow mode against real traffic first, migrate intent by intent starting with the highest-volume, lowest-risk cases, and keep the old decision-tree bot as a fallback for anything the new agent isn't confident about. Until your eval scores clear a bar you set in advance, not until it "feels ready."

Why big-bang cutover fails

A rules-based chatbot's decision tree is, in effect, a hand-built set of guardrails: it can only say things someone explicitly programmed it to say, which makes it limited but predictable. Replacing it outright with an LLM agent in one cutover throws away that predictability without necessarily replacing it with equivalent guardrails: the new agent can generate a wider range of responses, including wrong or off-brand ones, and if you haven't built the eval and confidence-scoring layer first, you find out about the gaps from customers, not from testing.

Step 1: inventory your existing bot's intents and baseline

Before writing any new prompts, catalog what the old bot actually handles: the list of intents/flows, their volume, and their current success rate (however you're already measuring it. Deflection rate, escalation rate, whatever exists). This is your baseline. Without it, you have no way to prove the new agent is actually better, only that it's different.

Step 2: run the new agent in shadow mode

Feed the LLM agent the same real inputs the old bot is receiving, but don't show its output to users yet. Compare its answers against the old bot's and against what a human reviewer would say is correct. This surfaces the failure modes before a customer ever sees them, and it's the closest thing to a real-world eval set you can build before launch.

Step 3: migrate intent by intent, highest volume and lowest risk first

Move traffic over one intent category at a time, starting with the intents that are both high-volume (so the win is meaningful) and low-risk (so a mistake is cheap to recover from). Order status lookups before anything involving account changes or money movement, for example. This mirrors the same sequencing logic we recommend for any legacy system modernization: incremental replacement beats a rewrite, because each step is independently verifiable.

Step 4: keep the old bot as a fallback, not delete it

Route to the old decision tree (or to a human) whenever the new agent's confidence is below your threshold, rather than forcing it to answer anyway. This is the single most effective way to avoid a visible regression during migration: the new agent only handles what it's actually good at yet, and the fallback covers the rest until eval scores justify expanding its scope.

Step 5: set rollback criteria in advance

Decide, before migration starts, what the eval threshold or escalation-rate ceiling is that would trigger rolling an intent back to the old bot. Deciding this after you've already seen a bad week of data is much harder to do objectively. You'll be negotiating with sunk cost, not evaluating results. For what a typical legacy-to-AI migration costs to scope and staff, see our modernization cost breakdown.

FAQ

Do you need to rewrite all your old chatbot flows before migrating? No: the intent-by-intent approach means you only need to have validated the flows you're actively migrating; the rest can keep running on the old bot indefinitely if that's the pace that fits your risk tolerance.

How do you measure if the new agent is actually better than the old bot? Against the baseline you captured in Step 1, using the same metrics: deflection rate, escalation rate, and ideally a sampled accuracy/quality score from human review, not just "it feels smarter."

Should the old rules-based bot be shut off immediately? No: keep it as the fallback path for low-confidence responses and unmigrated intents until the new agent has cleared your rollback thresholds across the board, not just in aggregate.

What's the biggest risk in chatbot-to-agent migrations? Losing the old bot's predictability faster than you build the new agent's guardrails: teams that skip the shadow-mode and confidence-threshold steps are the ones who find their failure modes via a customer complaint instead of a test.

Accelate migrates legacy chatbots to LLM agents intent by intent, with a fallback path and eval thresholds set before cutover, not a rewrite gamble.