How to test a RAG system before you trust it

How to evaluate a RAG system

Run six families of test against it before launch: golden cases with a known-correct answer, retrieval checks that the right record even reaches the model, abstention tests it is supposed to refuse, citation checks that trace every claim back to a retrieved record, determinism checks that the same question returns the same answer, and isolation tests that one tenant's data never surfaces in another's. Score retrieval separately from generation, gate the release on the results, and set each threshold against your own cost of a wrong answer, because no industry-standard passing grade exists.

The common alternative is shipping on impressions. Someone asks the demo twenty questions, likes nineteen of the answers, and calls it ready. That is how a system reaches real users with nobody knowing what it does when the answer is not in the corpus. For the mechanism behind a confident wrong answer, our guide to preventing AI hallucinations covers it. This page is the test plan.

Retrieval and generation fail differently, so measure them apart

A RAG system has two halves. The retriever finds records. The generator writes an answer out of what it found. A wrong answer can come from either half, and one end-to-end score tells you that something broke without telling you what.

The failure this hides is the expensive one. Retrieval returns nothing useful, the model fills the gap with a fluent paragraph, and the answer reads beautifully. Judge only the final text and a team can spend a month tuning prompts against a problem that lives in the chunking or the index.

The Ragas paper, "Ragas: Automated Evaluation of Retrieval Augmented Generation" (arXiv 2309.152171), splits evaluation along exactly that seam: retrieval effectiveness, generation fidelity, and overall answer quality. Its documentation gives the metrics names worth knowing on a vendor call. Context recall2 is the share of claims in a reference answer that the retrieved context supports, so a low score means the answer never entered the context window at all. Context precision3 measures whether relevant chunks were ranked above irrelevant ones. Faithfulness4 is the number of claims in the response supported by the retrieved context, divided by the total number of claims in the response.

TruLens frames the same split as the RAG Triad5: context relevance, groundedness, and answer relevance, and says the three together verify an application is "hallucination free up to the limit of its knowledge base". Read that limit closely. It is a claim about the knowledge base, not about the world. A system that faithfully cites a document nobody updated since 2023 scores well on all three and is still wrong.

One working rule makes the split pay off: tag every failing case as a retrieval failure or a generation failure before anyone attempts a fix. Early in a build, most RAG failures are retrieval failures wearing a generation costume.

The test plan

One row per test family. Build all six, run all six on every change, and decide in advance which failures block a release.

Golden cases

A golden case is a question with one answer somebody has already verified against the source record. Build the set from what people actually ask: support tickets, analyst requests, the questions the workflow owner answers by hand today. Store the record ID next to each answer, because that ID is what makes the retrieval test possible later. Then score wrong values rather than averages. A mean of 0.91 across a golden set hides which nine questions returned a plausible wrong number, and those nine are the only ones anybody will remember.

Abstention tests

Ask questions the corpus cannot answer and check that the system says so. Plausible-but-unanswerable is the difficult part to write, because a nonsense question is easy to refuse and proves nothing. The correct behaviour is a clear "I do not have that", not a hedge that still names a figure. Track two numbers, not one. Abstain precision asks whether every refusal was justified. A system that refuses answerable questions gets switched off by its own users, so count those too and watch the trade.

Citation grounding

Grounding only counts if the citation supports the claim. Sample answers from the golden and abstention runs and check each factual claim against the record it points to, one at a time, numbers digit by digit. The failure that survives casual review is a citation pointing at a genuine, relevant document that does not contain the figure in the answer. It looks correct in the interface. The automated version of this check compares each claim against the retrieved text before the answer is ever served, which is the job a verifier stage does.

Determinism

Ask the same question several times, on different days, and again after an index refresh. Two different revenue figures for one question in the same week destroys trust faster than a single wrong figure, because the user who sees it stops believing every other answer too. The causes are findable: sampling temperature, a reranker shuffling near-tied chunks, an index that changed underneath, a retry landing on a different model version. Fix the cause instead of dropping the temperature and hoping.

Isolation and leak tests

Where more than one client, team, or permission group shares an index, test that the wall holds. For each tenant, ask questions whose answers exist only in another tenant's records and confirm nothing comes back: not the value, not the document title, not a stray entry in the citation list. Repeat across every retrieval path and every embedding space, because a filter gets applied on one path and forgotten on another. Unlike a wrong number, nobody harmed by a leak is in the room to notice it.

There is no industry-standard passing score

Any accuracy percentage quoted without naming the test set behind it means nothing, vendor decks included. The threshold has to come from your side of the table.

NIST says as much about risk generally. Its AI Risk Management Framework (NIST AI 100-1, section 1.2.2) states that "while the AI RMF can be used to prioritize risk, it does not prescribe risk tolerance", and that "risk tolerance and the level of risk that is acceptable to organizations or society are highly contextual and application and use-case specific" (NIST AI 100-16).

The decision rule is short. Where a wrong answer can move money, quote a legal term, or reach a customer unreviewed, gate at zero wrong values and zero leaks and accept more abstention. Where the output is a draft a human edits, trade some abstention for coverage and gate on the citation audit instead.

One caveat, stated once: an eval suite is only as good as its questions, and a golden set written by the people who built the retriever will flatter it. Have the workflow owner write a share of it, and add every production failure to the set the week it happens.

A real production bar, from one system

On a grounded retrieval platform Ilayer built and operates for a $3.4B US real-estate investment firm, the gate holds abstain precision at 1.000, zero cross-tenant leaks across both embedding spaces, zero golden wrong-values, and zero determinism violations. It runs over 29.08M grounded chunks drawn from 1,314,631 documents, and every answer clears a seven-stage verifier that checks each claim against its source before it is served. A change that drops any of those numbers is reverted rather than patched.

Those figures come from one system, one corpus, and one set of questions written for it. They are not a benchmark and they do not transfer. What they show is that a zero-tolerance gate on the failures that matter most is achievable in production, which is worth knowing when a vendor tells you some error rate is unavoidable.

Where the gate goes in the release process

A suite that runs when someone remembers is a suite nobody runs. Wire the battery into the change process so every edit to a prompt, model version, chunk size, embedding model, reranker, or index build triggers all six families automatically, and so the thresholds you called unacceptable block the merge rather than producing a report read later. Keep the results per test family instead of rolling them into one score, or you lose the retrieval-versus-generation split you built the suite to get.

This is the layer teams cut when the demo goes well, and it is a large part of why AI pilots never reach production. Building the harness in-house is reasonable if you have the people. If you would rather not, it is what our LLM evals and reliability work delivers, including wrapping a pipeline that already exists.

None of this makes mistakes impossible. It makes them visible and traceable instead of silent, which is the only version of a mistake anyone can actually manage.