Your AI Project Is Six Months In and Stalled. Fix, Rebuild, or Kill?

Fix it if the problem is in the plumbing: data quality, chunking, retrieval, prompts, or missing evaluation. That covers most stalled projects and usually takes weeks, not months. Rebuild if the architecture is wrong for the problem, which shows up as every fix breaking something else. Kill it if users don't want the outcome, or if the accuracy the use case requires is beyond what the approach can reach. The expensive mistake is rebuilding a plumbing problem, because the new system inherits the same data and reproduces the same failures in fresh code.

What's the first question to ask?

Not "is the AI good enough." Ask: can anyone state a number?

If nobody can say what the accuracy is today, what it was three months ago, or what "good enough" would be, then the project isn't stalled on technology. It's stalled because there's no way to tell whether any change is progress.

This is the most common stall, and it's diagnostic rather than technical. Teams in this position have often been changing prompts and models for months, each change feeling like it helped, none of it measured. Effort without direction.

Establishing a baseline takes days. Until it exists, fix, rebuild, and kill are all guesses. Building an eval harness is the shortest path to one.

If the project never reached launch at all, the pattern is slightly different, and why most AI MVPs never reach production covers the readiness gaps that stall a build before it ships.

How do you tell a plumbing problem from an architecture problem?

Plumbing problems are local. Fix one and nothing else moves. Architecture problems are systemic, and the tell is that fixes trade against each other.

Signal	Plumbing	Architecture
Fixing one failure	Doesn't break others	Breaks something else
Failures	Cluster in one query type	Spread across everything
Adding documents	Works fine	Makes quality worse
The team's mood	"We know what's wrong, no time"	"Every fix opens two problems"
Latency	Acceptable, or fixable at one stage	Degrades as the system grows
Adding a feature	Contained change	Requires touching everything

That fourth row sounds soft and predicts well. A team that can name its problems has a plumbing problem. A team that describes a whack a mole pattern usually has a structural one.

The third row is the sharpest technical signal. If quality drops as the corpus grows, the retrieval approach doesn't scale to the corpus, and that's architectural.

What does the decision tree actually look like?

Work through in order. Stop at the first yes.

1. Do users actually want this outcome? Not the feature, the outcome. If usage is near zero after a real launch, and interviews show people don't want the job done, the problem isn't technical. Kill or repurpose. No amount of accuracy rescues an unwanted feature. If the doubt is about the approach rather than the demand, check whether the product needs RAG at all or whether fine-tuning fits better.

2. Does the use case require accuracy the approach can't reach? Some jobs need near certainty. If a wrong answer means legal exposure or patient harm, and the approach plateaus below what's required, more engineering won't close the gap. Narrow the scope to a safer subset, add human review, or stop.

3. Is there a measurement baseline? If no, build one first. Days of work. Everything below is unanswerable without it.

4. Does quality get worse as the corpus grows? If yes, that's architectural. Rebuild the retrieval layer, though usually not the whole system.

5. Do fixes break other things? If yes, architectural coupling. Rebuild the affected layer.

6. Everything else. Fix. This is where most projects land: stale data, poor chunk boundaries, no hybrid retrieval, weak prompts, no refusal policy, no monitoring. Weeks of work against a known list. The 48-hour audit checklist generates that list layer by layer.

What does each path cost, roughly?

Path	Typical scope	What you keep	Main risk
Baseline first	Days	Everything	None. This is a prerequisite
Fix	Weeks	The whole system	Fixing the wrong layer, if no baseline exists
Partial rebuild	Weeks to months	Data, UI, integrations, evals	Scope creeping into a full rebuild
Full rebuild	Months	Data and requirements only	Inheriting the original data problems
Kill	Immediate	Learnings, data pipeline, sometimes the UI	Sunk cost pressure preventing the decision

Partial rebuild is underused. Teams treat rebuild as all or nothing, when the usual honest finding is that retrieval is wrong and everything around it is fine. Replacing one layer while keeping ingestion, interface, and integrations is far cheaper than starting over.

Why is rebuilding usually the wrong instinct?

Because a rebuild feels like progress while changing the least important variable.

The code is rarely the problem. The data, the chunking strategy, and the absence of measurement are the problem, and a rebuild carries all three forward untouched. Six months later the new system fails the same way, and now there's less budget and less patience.

There's also a political dimension worth naming. A rebuild is easier to propose than a diagnosis, because it promises a clean slate and requires nobody to say what went wrong. That's exactly why it gets chosen more often than the evidence supports.

Rebuild when the evidence is structural. Not because the current system feels tired.

How do you make this decision defensible to your board?

Bring evidence, not sentiment.

• The current number and how it was measured
• The target number and where it came from
• The ranked list of findings, each with an effort estimate
• What each path costs and what it keeps
• The date you'll re-measure and decide again That last one matters most. Set a checkpoint, typically four to six weeks out, with a specific number attached. It converts an open ended commitment into a bounded one, which is usually what a nervous stakeholder actually needs.

If the checkpoint number isn't met, you've learned something real rather than spent another quarter hoping.

What vocabulary should you use in this conversation?

Term	Working definition
Baseline	The measured quality number today. Without it, no improvement is provable
Golden set	A fixed set of test queries with known correct outcomes
Plumbing problem	A local failure in data, chunking, retrieval, or prompts. Fixable without redesign
Architectural problem	A structural mismatch between approach and problem. Fixes trade against each other
Partial rebuild	Replacing one layer while keeping the rest. Usually the honest middle option
Scope narrowing	Reducing what the system claims to handle so it handles the rest reliably
Human in the loop	A person reviewing outputs before they reach the end user
Sunk cost	Money already spent. Irrelevant to the decision, though rarely irrelevant to the room
Checkpoint	A dated re-measurement with a pre-agreed threshold

The plumbing and architecture distinction is the most useful one to introduce early. It gives everyone in the room the same two categories to argue within, which tends to shorten the argument.

When this doesn't apply

• Projects under three months old. Six months of stall is a pattern. Six weeks is a normal build. Don't diagnose a project that's still being built.
• Research or exploratory work with no production target. Stalling is expected, and the framework's assumptions about users and accuracy don't hold.
• When the blocker is organisational. No data access, no domain expert time, no decision maker. The technical decision tree can't resolve a resourcing problem, and running it anyway produces a technical answer to a political question.
• Regulatory or procurement blocks. If the system works and can't be deployed for compliance reasons, that's a separate track entirely.
• Very small features. A single AI powered field in a form doesn't need a formal fix or kill process. Try, measure informally, move on.

FAQ

How do I know if we're rebuilding for the right reason? Write down the specific structural finding that makes a fix impossible, before deciding. If you can't name one in a sentence, such as "retrieval quality falls as the corpus grows past a certain size," it's likely frustration rather than evidence. Real architectural findings are specific and demonstrable.

Can we fix and rebuild at the same time? Rarely well. Running both splits attention and makes it impossible to attribute any improvement to either. A better pattern is fixing on a strict time box, typically four to six weeks, with a measurement at the end. If the number doesn't move, that's the evidence for a rebuild.

What if the vendor who built it says it just needs more time? Ask what number will change, by how much, and by when. A team with a real plan can answer specifically. "More time" without a target and a date is the same position that produced the current stall, and agreeing to it repeats the last six months. The questions to ask about any accuracy claim apply here too.

How much of a stalled project can be salvaged? More than teams expect. Data pipelines, integrations, interface, domain knowledge, and requirements usually survive any path except a full kill. The parts most often replaced are retrieval and chunking, which are a minority of the total build effort even though they cause a majority of the failures.

Is it ever right to kill a project that technically works? Yes. If users don't use it, or the value doesn't justify the running cost, a working system is still a failed investment. This is harder to admit than a technical failure, which is exactly why it drags on. Check usage numbers before accuracy numbers.

Who should make this call? The person accountable for the outcome, informed by someone technical who has no stake in either answer. The common failure is letting the team that built it decide whether to rebuild it. Independent evidence, whether internal or external, is what makes the decision defensible later.

---

Related reading

• What actually breaks in a RAG system? A 48-hour audit checklist — how to generate the findings list this decision needs
• Why most AI MVPs never reach production — the stall pattern before launch
• How do you know your AI feature is actually working? — establishing the baseline
• Your vendor says the AI is 95% accurate. What should you ask next? — holding a build team to a number
• Your AI answers confidently and wrongly: a diagnostic checklist — isolating the failing layer
• Does your AI product need RAG? A practical framework — revisiting the approach
• Build vs buy: should your SaaS build AI in-house or hire an agency? — who does the rebuild
• How to measure AI agent ROI — whether the value justifies continuing