The Symptom

The team described it the way users did. The bot sounded right but got things wrong. Ask it about a medication or a protocol detail and it would return a clean, confident, plausible answer that did not match the source documents. In a consumer app that is annoying. In a health product it destroys trust, and the team could feel adoption stalling because clinicians stopped believing the answers.

The root cause

Three things were quietly compounding, and none of them were the LLM.

• Chunking was severing context. Chunks of a fixed size split clinical passages before they finished, so the retrieved fragment often lacked the qualifier that changed its meaning, the "except in patients with" clause that flips an answer.
• Pure vector search missed exact terms. Semantic similarity is good for concepts and weak on exact tokens like drug names, dosages, codes, and abbreviations. The passage a clinician needed was frequently not in the top results.
• No grounding, so the model filled gaps by guessing. With weak retrieval and no way to abstain, the LLM did what LLMs do. It produced a fluent answer anyway, and fluency made the wrong answers more convincing, not less.

What We Changed

We rebuilt the retrieval path, not the model.

• Chunking that keeps whole clinical passages together instead of cutting on a character count.
• Hybrid retrieval keyword and semantic search running together, so exact terms and concepts both land.
• A reranking step over the candidate passages, so the most relevant evidence reaches the model first rather than whatever scored highest on raw similarity.
• Grounded answers with abstention. The bot answers from retrieved evidence and cites it, and it says it does not have enough information instead of inventing an answer.

The Result

The behaviour change was the point. The bot stopped producing confident answers it could not support. Answers now trace back to a specific source passage the user can check, and when the evidence is not in the corpus, it declines rather than guesses. Clinician trust, the thing that had stalled, was the metric the team actually cared about, and it moved because the failure mode that broke it was gone.

The Lesson

In a domain where a wrong answer is costly, retrieval quality is the product. A bigger or newer model cannot rescue bad retrieval. If the right passage never reaches the model, all a stronger model does is write a more convincing wrong answer. Hybrid search, reranking, and honest abstention beat a model upgrade almost every time, and they cost less.

When This Doesn't Apply

If your corpus is small, uniform, and low risk, a FAQ bot over fifty tidy documents, a naive vector setup is fine and this rebuild is too much. The payoff scales with two things: how much your terminology depends on exact matches, and how expensive a wrong answer is. Medicine maxes out both.

FAQ

Wouldn't a better model have fixed this?
No. The wrong answers came from the model never seeing the right passage. A stronger model with the same bad retrieval produces the same wrong answers, just more fluently.

What is hybrid retrieval?
Running keyword search and semantic (vector) search together. Keyword catches exact terms like drug names and codes; semantic catches concepts and paraphrase. Neither alone is enough for clinical text.

How does abstention help accuracy?
It converts a class of confident-wrong answers into honest "I don't know" responses. In a health context, a decline is safe; a fluent wrong answer is not.

Isn't reranking slow?
It adds a step, but it runs on a small candidate set, and the latency cost is minor next to the trust cost of surfacing the wrong passage.