Guide
How to evaluate a RAG system before you trust it
RAG is short for retrieval-augmented generation. Six test families, and the threshold to gate each one at. No industry-standard passing score exists, so here is how to set your own.
Published Updated 9 min read
How to evaluate a RAG system
Evals are tests for a large language model's answers. Run six families of them before launch:
- Golden cases with a known-correct answer.
- Retrieval checks that the right document even reaches the model.
- Abstention tests on questions it should refuse. The right answer is "I do not have that".
- Citation checks that trace every claim back to a retrieved document.
- Determinism checks that the same question returns the same answer.
- Isolation tests that one tenant's data never surfaces in another's.
Score retrieval apart from generation. Gate the release on the results. Set each threshold against your own cost of a wrong answer. No industry-standard passing grade exists, so nobody can set it for you.
The common alternative is shipping on impressions. Someone asks the demo twenty questions. They like nineteen of the answers and call it ready. That is how a system reaches real users with nobody knowing what it does when the answer is not in the corpus. For the mechanism behind a confident wrong answer, read our guide to preventing AI hallucinations. This page is the test plan.
Retrieval and generation fail differently, so measure them apart
A RAG system has two halves. Your documents get split into searchable passages first. The retriever finds the ones that look like they answer the question. The generator writes an answer out of what it found. A wrong answer can come from either half. One end-to-end score tells you something broke, and never tells you what.
The failure this hides is the expensive one. Retrieval returns nothing useful. The model fills the gap with a fluent paragraph, and the answer reads well. Judge only the final text and a team can spend a month tuning prompts. The problem was in the chunking or the index the whole time.
The Ragas paper, "Ragas: Automated Evaluation of Retrieval Augmented Generation" (arXiv 2309.15217), splits evaluation along exactly that seam. The three parts are retrieval effectiveness, generation fidelity, and overall answer quality. Its docs name the metrics worth knowing on a vendor call. Context recall is the share of claims in a reference answer that the retrieved context supports. A low score means the answer never entered the context window at all. Context precision measures whether relevant chunks were ranked above irrelevant ones. Faithfulness counts the claims in the response that the retrieved context supports. Divide that by the total number of claims in the response and you have the score.
TruLens frames the same split as the RAG Triad: context relevance, groundedness, and answer relevance. The three together, it says, verify an application is "hallucination free up to the limit of its knowledge base". Read that limit closely. It is a claim about the knowledge base, not about the world. A system can faithfully cite a document nobody updated since 2023. It scores well on all three and is still wrong.
One working rule makes the split pay off. Tag every failing case as a retrieval failure or a generation failure before anyone attempts a fix. Early in a build, most RAG failures are retrieval failures wearing a generation costume.
The test plan
One row per test family. Build all six. Run all six on every change. Decide in advance which failures block a release.
| Test family | What it measures | How you build the set | What a failure looks like | Where to set the gate |
|---|---|---|---|---|
| Golden cases | Whether the answer matches the one known-correct value | Take 50 to 100 real questions per agent from tickets and analyst requests. A human verifies each answer against the source document. | The answer names a plausible figure. It does not match the source. | Zero wrong values. One wrong number on a golden case blocks the release. |
| Retrieval recall | Whether the answer document reaches the model at all, before the model writes anything | Reuse the golden questions. Store the ID of the document holding each answer. Then check whether that ID appears in the top k retrieved. | The right document sits at rank 40. It never enters the context window. | Fix k first. Then require the answer document inside the top k for a stated share of golden questions. Below that, repair retrieval, not the prompt. |
| Abstention | Whether the system says it does not know when the corpus cannot answer | Write questions that sound answerable and are not. Ask about a property you do not own, a quarter with no data, a field nobody ever captured. | The system invents a confident figure, or hedges and still states a number. | Abstain precision at 1.000, every refusal justified. Count the answers given on unanswerable questions too, and gate those at zero. |
| Citation grounding | Whether every claim in the answer traces to a document that actually supports it | Sample answers from the golden and abstention runs. Check each claim against the cited document, digit by digit for numbers. | A citation points at a real, relevant document. That document does not contain the figure quoted. | Gate at zero uncited numeric claims. On the audited sample, no citation may fail to support its claim. |
| Determinism | Whether the same question gets the same answer twice | Run the golden set again at production settings, on different days. Run it again after an index refresh. | The same question returns two different revenue figures in one week. | Gate at zero determinism violations on the golden set. Measure at production settings. |
| Isolation and leaks | Whether one tenant, team or permission group can reach another's data | Per tenant, ask questions whose answers exist only in another tenant's documents. Repeat across every retrieval path and every embedding space. | A value, a name or a document title from another tenant appears in the answer or the citation list. | Zero leaks. A single leak is a stop-ship, not a bug ticket. |
Golden cases
A golden case is a question with one answer somebody has already verified against the source. Build the set from what people actually ask: support tickets, analyst requests, the questions the workflow owner answers by hand today. Store the document ID next to each answer. That ID is what makes the retrieval test possible later.
Score wrong values, not averages. A mean of 0.91 across a golden set hides which nine questions returned a plausible wrong number. Those nine are the only ones anybody will remember.
Abstention tests
Ask questions the corpus cannot answer, and check that the system says so. The hard part is writing questions that sound answerable. A nonsense question is easy to refuse and proves nothing. Correct behaviour is a clear "I do not have that". A hedge that still names a figure is not.
Track two numbers. Abstain precision asks whether every refusal was justified. Then count the answerable questions the system refused. A system that refuses answerable questions gets switched off by its own users, so watch the trade.
Citation grounding
Grounding only counts if the citation supports the claim. Sample answers from the golden and abstention runs. Check each factual claim against the document it points to, one at a time, numbers digit by digit. The failure that survives casual review is quieter. The citation points at a genuine, relevant document, and that document does not contain the figure in the answer. It looks correct in the interface. The automated version compares each claim against the retrieved text before the answer is served. That is the job a verifier stage does.
Determinism
Ask the same question several times, on different days, and again after an index refresh. Two different revenue figures for one question in the same week destroys trust faster than a single wrong figure. The user who sees it stops believing every other answer too. The causes are findable: sampling temperature, a reranker shuffling near-tied chunks, an index that changed underneath, a retry landing on a different model version. Fix the cause. Do not drop the temperature and hope.
Isolation and leak tests
Where more than one client, team or permission group shares an index, test that the wall holds. For each tenant, ask questions whose answers exist only in another tenant's data. Confirm nothing comes back: not the value, not the document title, not a stray entry in the citation list. Repeat across every retrieval path and every embedding space. A filter gets applied on one path and forgotten on another. Unlike a wrong number, nobody harmed by a leak is in the room to notice it.
There is no industry-standard passing score
Any accuracy percentage quoted without naming the test set behind it means nothing. That goes for vendor decks too. The threshold has to come from your side of the table.
NIST says as much about risk generally. Its AI Risk Management Framework (NIST AI 100-1, section 1.2.2) states that "while the AI RMF can be used to prioritize risk, it does not prescribe risk tolerance". It adds that "risk tolerance and the level of risk that is acceptable to organizations or society are highly contextual and application and use-case specific" (NIST AI 100-1).
The decision rule is short. Where a wrong answer can move money, quote a legal term or reach a customer unreviewed, gate at zero wrong values and zero leaks. Accept more abstention as the price. Where the output is a draft a human edits, trade some abstention for coverage. Gate on the citation audit instead.
One caveat, stated once. An eval suite is only as good as its questions, and a golden set written by the people who built the retriever will flatter it. Have the workflow owner write a share of it. Add every production failure to the set the week it happens.
A real production bar, from one system
On a grounded retrieval platform Ilayer built and still operates, the gate holds abstain precision at 1.000 and zero cross-tenant leaks across both embedding spaces. It also holds zero golden wrong-values and zero determinism violations. The platform runs over 29 million grounded chunks drawn from more than 1.3 million documents. Every answer clears a seven-stage verifier that checks each claim against its source before it is served. A change that drops any of those numbers is reverted, not patched.
Those figures come from one system, one corpus, and one set of questions written for it. They are not a benchmark and they do not transfer. What they show is narrower. A zero-tolerance gate on the failures that matter most is achievable in production. That is worth knowing when a vendor tells you some error rate is unavoidable.
Where the gate goes in the release process
A suite that runs when someone remembers is a suite nobody runs. Wire the battery into the change process. Every edit to a prompt, model version, chunk size, embedding model, reranker or index build should fire all six families on its own. The thresholds you called unacceptable should block the merge. A report somebody reads later is not a gate. Keep results per test family instead of rolling them into one score. Roll them up and you lose the split between retrieval and generation that you built the suite to get.
This is the layer teams cut when the demo goes well. It is a large part of why AI pilots never reach production. Building the harness in-house is reasonable if you have the people. If you would rather not, it is what our LLM evals and reliability work delivers, including wrapping a pipeline that already exists.
None of this makes mistakes impossible. It makes them visible and traceable instead of silent. That is the only version of a mistake anyone can manage.
Questions people ask
How do I measure whether our RAG system is accurate?
Measure retrieval and generation apart. A wrong answer can come from either half, and the repair differs in each case. Score retrieval on one question: does the document holding the answer ever reach the model? Score generation on another: is every claim in the response supported by what was retrieved? Then run a fixed set of questions whose answers a human has already verified against the source. Count wrong values. Do not report an average. One wrong number on a question you already knew the answer to matters more than a good mean.
What is a good accuracy score for a RAG system?
No industry-standard passing score exists. An accuracy percentage quoted without naming the test set behind it means nothing. NIST's AI Risk Management Framework says plainly that it does not prescribe risk tolerance. Acceptable risk, it says, is highly contextual and application and use-case specific. So set the threshold from what a wrong answer costs in the workflow. A system quoting lease terms to an investor should gate at zero wrong values. A drafting assistant a human edits can live with less. What carries across use cases is the shape of the gate, not the number.
How do I test a RAG system before it goes live?
Build the test sets before launch. You need five things. Golden questions with known answers. Unanswerable questions the system should refuse. Citation audits. Repeat runs of the same question, and cross-tenant questions that probe isolation. Run all of them on every change to a prompt, a model version, a chunking rule or an index build. Block the release on the failures you decided are unacceptable. Do not review them afterwards. Testing by hand in a chat window finds the failures somebody thought of. The ones that hurt in production are the ones nobody did.
Keep reading
- 4 August 2026How to stop AI hallucinationsFour mechanisms that cut invented answers, the number to demand for each, and the eight questions that expose a vendor who has never measured one.
- 4 August 2026Why AI projects fail between the demo and the deadlineFive failure patterns, the symptom each one shows first, and an eight-question checklist to hold your own pilot against at week nine.