What Actually Breaks in a RAG System? A 48-Hour Audit Checklist

RAG systems break at seven layers, and almost never at the layer people blame. In order: ingestion, chunking, retrieval, reranking, generation, grounding, and monitoring. Teams usually blame the model, which is the sixth thing worth checking, not the first. The fastest way to find the real problem is to walk the layers in order and stop at the first one that fails, because a failure upstream makes every downstream measurement meaningless. Most of what looks like a model problem turns out to be a chunking or retrieval problem, and those are days of work rather than a rebuild.

Why do teams misdiagnose RAG failures?

Because the symptom always appears at the end. The user sees a wrong answer, so the wrong answer looks like a generation problem.

But a wrong answer can be produced by a document that never got ingested, a chunk that split a table in half, a retriever that ranked the right passage twelfth, or a prompt that let the model improvise. All five produce the identical symptom.

This is why "let's try a better model" is the most common first move and one of the least effective. If the correct passage never reached the context window, no model will fix it. The diagnostic checklist for confidently wrong answers walks the same logic from the symptom end.

What are the seven layers, and what does each failure look like?

Layer	What it does	Symptom when it fails	Cheapest test
Ingestion	Gets source documents into the system	Answers about "missing" content; system swears a document doesn't exist	Count documents in source vs count in index
Chunking	Splits documents into retrievable pieces	Answers that are half right, or cut off mid fact	Read 20 random chunks. Do they make sense alone?
Retrieval	Finds candidate chunks for a query	Right answer exists but never surfaces	Search a query where you know the source. Is it in the top 10?
Reranking	Reorders candidates by true relevance	Correct chunk is retrieved but ranked low	Compare rank before and after reranking on 20 queries
Generation	Writes the answer from retrieved context	Correct context, wrong or vague answer	Paste the retrieved chunks into the model manually. Does it answer correctly?
Grounding	Ties each claim to a source	Confident answers with wrong or missing citations	Check whether the cited span actually supports the claim
Monitoring	Detects change over time	Quality drops and nobody notices for weeks	Ask when quality was last measured. If nobody knows, this layer is absent

Work top to bottom. A chunking problem will corrupt every retrieval measurement you take, so fixing retrieval before chunking wastes the effort.

Two layers have enough depth to warrant their own treatment. Grounding failures are covered in why citation accuracy is harder than retrieval accuracy, and the monitoring layer is covered in how to know your AI feature is actually working.

What does the 48-hour audit actually look like?

Two days is enough to locate the failing layer and size the fix. It is not enough to fix everything, and an audit that promises that is selling something else.

Hours 0 to 8: Ingestion and corpus reality

Start with what's actually in the index, because this is where the most embarrassing findings live.

• How many source documents exist? How many are in the index? The gap is the finding.
• When did the last successful sync run? Check for silent failures.
• Are there near duplicate documents, especially versioned ones like policy v1 through v4?
• Did tables, PDFs, and scanned files parse correctly, or did they enter as noise?
• Is there content the system should never answer from, such as drafts or internal notes? A common finding here: the index was built once during development and never updated. The system is answering from a snapshot that's months old.

Hours 8 to 20: Chunking and retrieval

Pull 20 real user queries where you know the correct source document.

• For each, is the correct chunk in the top 10 retrieved results? Record the rank.
• Read 20 random chunks with no context. Can you tell what each one is about?
• Do chunks split tables, lists, or procedures across boundaries?
• Does the system handle exact term queries, such as part numbers, drug names, or error codes? That last one catches a specific and very common failure. Pure semantic search is weak on exact strings, because an embedding of a part number sits near other part numbers rather than near the exact match. Hybrid retrieval, which combines keyword search with semantic search, is the standard fix.

Hours 20 to 32: Generation and grounding

Now that you know retrieval works, you can test generation honestly.

• Take queries where the correct chunk was retrieved. Was the answer still wrong? That isolates a generation problem.
• Does every factual claim carry a citation?
• Does the cited source actually support the claim, or is it merely on the same topic?
• Ask an out of scope question. Does the system decline, or does it improvise? The out of scope test is the one clients underestimate. A system that answers questions it was never built for will eventually answer one that costs money.

Hours 32 to 44: Evals, monitoring, and cost

• Does a golden set of test queries exist? If not, that's the largest gap in the system.
• Can anyone state the current accuracy number? A feeling is not a number.
• What happens on deploy? Is there any gate, or does code ship untested?
• What does a single query cost, broken into embedding, retrieval, and generation?
• Is the system re-embedding documents that haven't changed? That last question regularly uncovers meaningful waste, because naive sync pipelines re-embed the entire corpus on every run rather than only what changed.

Hours 44 to 48: Findings and sizing

The deliverable is a ranked list. For each finding: the layer, the evidence, the user visible impact, the fix, and a rough effort band. Ranked by impact divided by effort, not by how interesting the problem is.

Which findings are cheap to fix, and which are structural?

Finding	Typical effort	Structural?
Stale index, broken sync	Small	No
Missing citations in output	Small	No
No out of scope refusal policy	Small	No
Re-embedding unchanged documents	Small to medium	No
Chunk boundaries splitting facts	Medium	No
No hybrid retrieval on exact terms	Medium	No
No eval harness or golden set	Medium	No, but it blocks everything else
Wrong chunking strategy for the document type	Medium to large	Sometimes
Retrieval architecture unsuited to the corpus	Large	Yes
Use case that RAG cannot serve	Rebuild or stop	Yes

Most audits land in the top half of that table. The genuinely structural findings are the minority, which is worth saying plainly to a client who arrived expecting to hear their system is unsalvageable. If the findings do come back structural, the fix, rebuild, or kill decision covers how to weigh the options.

What vocabulary should you use in an AI system audit?

Term	Working definition
Ingestion pipeline	The process that pulls source documents into the system and keeps them current
Chunk	A piece of a document sized to be retrieved and read independently
Chunk boundary	Where one chunk ends and the next begins. Bad boundaries split facts in half
Recall@k	Whether the correct document appears in the top k retrieved results
Dense retrieval	Semantic search using embeddings. Strong on meaning, weak on exact strings
Sparse retrieval / BM25	Keyword search. Strong on exact terms, blind to paraphrase
Hybrid retrieval	Running both and merging results. The standard fix for exact term failures
Reranking	A second pass that reorders retrieved candidates by true relevance
Grounding	The property of a claim being traceable to a real source
Out of scope handling	Correctly declining questions the system wasn't built to answer
Golden set	A fixed set of test queries with expected outcomes, used as the quality baseline
Drift	Quality degrading over time with no code change

Using these precisely is what separates an audit from an opinion. "Your Recall@10 is fine, your reranking isn't" is a finding. "Retrieval seems off" is a guess.

When this doesn't apply

This checklist assumes a retrieval backed system in or near production. It fits poorly in several cases.

• Prototypes and demos. A two week old prototype hasn't earned an audit. Ship it, get real usage, then look.
• Systems with no retrieval step. Pure generation, classification, or extraction features fail differently. Layers one through four don't exist.
• Very small corpora. Under a few hundred documents with no near duplicates, retrieval rarely fails. Focus on grounding and out of scope handling instead.
• Agent systems with tool calling. RAG is one component. Tool selection, error handling, and loop control are separate failure surfaces this checklist doesn't cover.
• When the real problem is product, not engineering. If users don't want the feature, a perfectly grounded answer to a question nobody asks is still a failed project. It's worth confirming whether your product needs RAG at all before auditing how well it runs.

FAQ

How long does a RAG audit really take? Two focused days is enough to locate the failing layer and size the fixes, provided you have access to the index, real user queries, and logs. Without logs it takes longer, because you have to generate representative queries rather than sample them. Fixing what's found is separate work.

Do I need to hand over my data for an audit? Not necessarily. Much of the checklist runs on structure rather than content: document counts, chunk boundaries, retrieval ranks, prompt design, and deploy process. Sensitive corpora can be audited with a redacted sample or with the client running the queries and sharing results.

What's the single most common finding? The absence of any quality baseline. Most stalled systems have no golden set and no recorded accuracy number, which means nobody can prove whether any change helped. This makes every other problem harder to fix, because there's no way to measure progress against it.

Should I audit before or after choosing a vendor to fix it? Before, where possible. An audit performed by the team that will also quote for the fix has an obvious incentive problem. If the same team does both, ask for the raw evidence behind each finding, not just the conclusions, so the recommendations can be checked independently.

Can I run this checklist myself? Yes, and the first two sections especially. Counting documents, checking sync recency, and reading 20 random chunks require no specialist tooling. The retrieval and grounding sections need more instrumentation, but the ingestion findings alone often explain the complaint that prompted the audit.

Is it ever cheaper to rebuild than to fix? Sometimes, though less often than teams assume. Rebuild is justified when the retrieval architecture is wrong for the corpus or when the use case doesn't suit retrieval at all. Rebuilding because of chunking or prompt problems is expensive and usually reproduces the same issues in new code.

---

Related reading

• Your AI answers confidently and wrongly: a diagnostic checklist — the same problem approached from the symptom
• Why is citation accuracy harder than retrieval accuracy in RAG? — the grounding layer in depth
• How do you know your AI feature is actually working? — the monitoring layer in depth
• Your vendor says the AI is 95% accurate. What should you ask next? — verifying a quality claim
• Your AI project is six months in and stalled. Fix, rebuild, or kill? — what to do with the findings
• Does your AI product need RAG? A practical framework — deciding before building
• RAG vs fine-tuning: which one should you choose? — when retrieval isn't the right approach