The Most Common RAG Failure Modes, and How to Catch Them Before Launch

Quick answer

Most RAG failures aren't generation failures. They're retrieval failures wearing a hallucination costume. The model answered badly because it got the wrong chunks, missing chunks, or chunks with no clear signal that they were irrelevant, not because it invented something out of nowhere. Catch these before launch with a retrieval-only eval pass, tested separately from your end-to-end answer eval, because a good final answer can hide a retrieval process that only got lucky.

Failure mode 1: the wrong chunk gets retrieved

Near-duplicate content (two similar policy versions, two product tiers with overlapping language) or an ambiguous query can pull back a chunk that's topically close but factually wrong for the specific question asked. This is the most common failure and the hardest to notice from the outside, because the answer often still reads fluently, it's just wrong.

Catch it: build a retrieval-only eval set (queries paired with the specific chunk ID that should be retrieved) and check retrieval accuracy independent of whether the final generated answer happens to sound right.

Failure mode 2: right document, bad chunking

The correct answer exists in the source material, but it's split across a chunk boundary, so no single retrieved chunk contains the complete answer. The model then either answers with half the information or fills the gap with a plausible-sounding guess.

Catch it: test cases specifically drawn from answers that span multiple sentences or sections in the source document, and check whether your chunking strategy keeps them together.

Failure mode 3: stale index

The source document was updated, but the embeddings/index weren't refreshed, so the agent confidently serves outdated information with no way to know it's stale. This is a data-freshness problem, not a model problem, and it's invisible until someone notices the agent quoting last month's numbers.

Catch it: monitor time-since-last-reindex as an operational metric, not just an eval metric, and set an alert threshold based on how often your source content actually changes.

Failure mode 4: no result found, but the model answers anyway

When retrieval genuinely returns nothing relevant, a model that hasn't been explicitly instructed and tested for this case will often answer from its general training knowledge instead of saying "I don't know" or escalating. Which is exactly the scenario RAG was supposed to prevent.

Catch it: deliberately include out-of-scope questions in your eval set and verify the agent declines or escalates rather than guessing. This is a testable, gateable behavior, not a hope.

Failure mode 5: correct chunk retrieved, but buried

The right chunk is in the retrieved set, but it's ranked low or surrounded by irrelevant results, and the model doesn't weight it properly against the noise. A known "lost in the middle" pattern where information isn't used reliably regardless of position in a long context.

Catch it: test with a deliberately noisy retrieval set (correct chunk plus several distractors) rather than only testing with a clean, ideal retrieval set: production retrieval is never as clean as your happy-path test.

Building the eval habit

All five of these are testable independently, before launch, if you build a retrieval-only eval layer separate from your end-to-end answer eval. See our broader guide to production observability and eval tooling for how that layer should be structured. And if you're still deciding whether your product needs RAG at all, that upstream question is covered in our RAG decision framework; if the failures you're seeing look more like a behavior problem than a knowledge problem, see RAG vs. fine-tuning for accuracy for how to tell the difference.

FAQ

How do you test RAG retrieval quality separately from the final answer? Build an eval set of queries paired with the specific correct source chunk, and measure whether retrieval returns that chunk in its top results: independent of whether the model's final answer happens to read correctly.

What causes a RAG system to hallucinate even with retrieval enabled? Usually one of the five failure modes above: wrong chunk retrieved, the answer split across a chunk boundary, a stale index, no relevant result found but the model answers anyway, or the correct chunk being present but buried among irrelevant ones.

How often should you re-index or refresh embeddings? As often as your source content actually changes (daily for fast-moving content, weekly or monthly for stable reference material) but the schedule should be a deliberate decision tied to your content's update cadence, not a default left unexamined.

Is chunk size the most important RAG tuning variable? It matters, but it's one variable among several. Retrieval ranking, index freshness, and how the model is instructed to handle "no good match found" cases all matter as much or more, and tuning chunk size alone won't fix the other four failure modes.

Accelate builds a retrieval-only eval layer into every RAG project before launch, so failure modes get caught in testing instead of in a customer's inbox.