How to Judge a RAG System
RAG evaluation, one metric at a time: context precision and recall judge the retrieval (did it find the right pages), faithfulness and response relevancy judge the generation (did the answer stick to those pages and address the question). Each with its plain-English meaning, its formula, and a worked example, grounded in the Ragas metrics.
Part eight of the RAG series. A wrong RAG answer has two possible culprits, and one number cannot tell them apart. Evaluation splits like grading an open-book exam: retrieval quality (context precision, did it bring back relevant pages and rank them first; context recall, did it miss any needed page) and generation quality (faithfulness, did the answer only say what the pages support; response relevancy, did it address the question asked). This defines a classic four-metric starting set from Ragas, each with its formula and a worked example, explains which need a ground-truth reference and which use an LLM as judge, and why an LLM-judge score is a moving target. Everyday analogies, a hand-drawn diagram, and a visualizer per metric.
The previous part was about operating RAG in production: keeping the index true to a moving corpus, and tracing every answer. It kept naming metrics, faithfulness, mean reciprocal rank, hallucination rate, without defining them. This part defines the ones that matter, one at a time. If you have not read the earlier parts, start with the field guide; this one assumes you know what retrieval and generation are.
Think about grading an open-book exam. A student hands in an answer, and it is wrong. There are two completely different reasons that could have happened, and they need different fixes. Either the student opened to the wrong pages (so they never had the information), or they had the right pages open and still wrote a bad answer (ignored them, or made something up). "The answer was wrong" does not tell you which. You have to grade the two things separately: did they find the right pages, and did they use them well.
A RAG system is exactly this open-book exam, run thousands of times a day. It has a retrieval step (find the pages) and a generation step (write the answer from them). So a single accuracy score is not enough for debugging, the same way "wrong" is not useful feedback to a student. A useful first dashboard has four numbers, two for each half. A very good starting point is the four-metric set popularized by Ragas, and that is what this article defines. (Production evaluation grows past these four, into task correctness, citation quality, safety, latency, and cost, but this quadrant is where you start.)
The two halves, and why one number lies
Here is the split to hold onto for the whole article. Retrieval quality asks whether the right pages were put in front of the model. Generation quality asks whether the model then wrote a good answer from those pages. A low final score could come from either, and the fix is different: a retrieval failure means work on chunking, embeddings, or reranking; a generation failure means work on the prompt or the model. So we grade retrieval with two metrics (context precision and context recall) and generation with two (faithfulness and response relevancy). Each of the next four sections is one metric: what it means in plain words, its formula, and a worked example.
One thing to know up front: most of these are judged by another language model reading the answer, not by a simple string match, because "does this claim follow from this passage" is a judgment call. That makes the scores capable but noisy, a point we return to at the end.
Context precision: is what came back actually on-topic, and ranked first
Picture a research assistant who hands you a stack of pages for your question. Precision asks: of that stack, how much is genuinely about your question versus padding, and did they put the useful pages on top? A stack that is half junk, or that buries the one good page at the bottom, scores low even if the good page is in there somewhere.
Context precision measures how much of the retrieved context is relevant, and it rewards ranking the relevant chunks near the top. Ragas computes it as the mean of precision@k over the retrieved chunks: precision@k is the fraction of the top k chunks that are relevant, and the metric averages those, counting only the ranks where a relevant chunk actually sits. Ranking matters because retrieval runs on a limited context budget, and higher-ranked chunks are the ones most likely to make it into the prompt at all.
Worked example. Four chunks come back, and in ranked order chunks 1, 3, and 4 are relevant while chunk 2 is noise (relevance pattern 1, 0, 1, 1). Precision@1 is 1/1 = 1.0, precision@3 is 2/3 = 0.67, precision@4 is 3/4 = 0.75, and rank 2 is skipped because that chunk is not relevant. Averaging over the three relevant ranks: (1.0 + 0.67 + 0.75) / 3 = 0.81. If instead the only relevant chunk were buried at rank 4 (pattern 0, 0, 0, 1), precision@4 is 1/4 = 0.25 and that is the only relevant rank, so the score collapses to 0.25 even though the same chunk was technically retrieved. Same recall, very different precision, because position is the whole point.
Four chunks come back, ranked. Three are relevant (green), one is noise (grey). Precision asks how much of the stack is on-topic.
Context recall: did it leave a needed page on the shelf
Precision's opposite question. Recall does not care about junk in the stack; it asks whether any important page was left on the shelf. Even one missing key page means the answer cannot be complete, no matter how clean the rest of the stack is.
Context recall measures how much of the information actually needed to answer was present in the retrieved context. Because "what was needed" has to come from somewhere, context recall needs some notion of ground truth, a reference answer or reference contexts: Ragas breaks the reference into claims and checks what fraction of those claims are supported by the retrieved context. The formula is the number of reference claims found in the context divided by the total number of reference claims. (In the current Ragas API, context precision compares retrieved chunks against a reference answer; when no reference is available, the closely related context-utilization metric judges them against the generated response instead, and legacy APIs also expose reference-free precision variants. Context recall is the one that genuinely cannot do without a reference.)
Worked example. The correct answer to "what does the return policy cover and how long do I have" contains 5 distinct facts: the 30-day window, the receipt requirement, the exclusion for opened software, the store credit option, and the exchange rule. The retrieved context contains 4 of them but never mentions the store-credit option. Context recall is 4/5 = 0.80. That missing fact is invisible to precision (the four retrieved chunks were all relevant, so precision looks great) and it is invisible to the final answer's fluency. Only recall catches it, and it catches the most dangerous failure in RAG: a confident answer that is silently incomplete.
The correct answer needs 5 distinct facts. Recall asks how many of them retrieval actually brought back.
Faithfulness: did the answer stick to the book
Now the generation half. Faithfulness is the open-book rule itself: the answer may only say things the retrieved pages actually support. If the model states something the pages do not back, for faithfulness it counts as unsupported (ungrounded), even if that claim happens to be true in the outside world, because faithfulness measures consistency with the retrieved context, not truth against the whole world.
Faithfulness measures the fraction of the answer's claims that are supported by the retrieved context. Ragas computes it by having a model break the answer into individual claims, then checking each claim against the context: faithfulness equals the number of supported claims divided by the total number of claims. In the docs' notation, F = |supported| / |total statements|.
Worked example. The generated answer makes 5 claims: "returns accepted within 30 days" (in the context), "receipt required" (in the context), "opened software excluded" (in the context), "refunds go to the original card in 5 business days" (NOT in the context, the model guessed the 5 days), and "exchanges allowed in store" (in the context). Four of the five claims are grounded; one was invented. Faithfulness is 4/5 = 0.80. Notice this diagnoses a different failure from retrieval quality: even if retrieval brought back everything needed, the generator can still add an unsupported claim. That is why you measure it separately.
The generated answer makes 5 claims. Faithfulness checks each one against the retrieved context.
Response relevancy: did it answer the question that was asked
The last metric catches a sneakier failure: an answer that is perfectly faithful and perfectly sourced, but does not actually address what you asked. Think of a student who, asked about the return window, writes three correct paragraphs about shipping. Every sentence is true and from the book; none of it answers the question.
Response relevancy (historically called answer relevancy, and still the name most people use) measures how directly the answer addresses the original question. Ragas computes it in a reference-free way: it asks a model to generate several questions that the given answer would be a good response to, embeds each of those questions, and measures their average cosine similarity (a score for how close two pieces of text are in meaning) to the original question. If the answer is on point, the questions it implies look just like the real question; if the answer wandered, they drift away. The score is the average of those similarities: AR = (1/m) times the sum of sim(Q, q_i).
Worked example. The question is "how long do I have to return a defective laptop." A tight answer ("you have 30 days from delivery to return a defective laptop") makes a model generate implied questions like "what is the return window for a defective laptop," which sit very close to the original, giving a high similarity. A padded answer that mostly discusses shipping makes the model generate implied questions about shipping times, which sit far from the original, dragging the average down. The exact numbers here (say 0.95 versus 0.60) are illustrative, not computed outputs; what is real is the gap. Same faithfulness (both stuck to the context), very different relevancy.
The question: 'how long do I have to return a defective laptop?' Relevancy asks whether the answer actually addresses this.
What actually runs, and the catch
Ragas is the most common open-source library for these four metrics, and it names more (context entities recall, noise sensitivity for robustness to irrelevant retrieved chunks, and others), but precision, recall, faithfulness, and relevancy are the load-bearing quadrant. TruLens, DeepEval, and Arize Phoenix compute a similar set; the cloud vendors wrap their own versions too. The mechanics are what matter more than the brand: two metrics for retrieval, two for generation.
The catch, and it is a big one: the commonly used versions of these metrics lean on evaluator models. Precision, recall, and faithfulness use a language model to make the semantic judgments (is this chunk relevant, is this claim supported); response relevancy uses a language model to generate the implied questions and then embeddings to score how close they are to the real one. That makes the scores flexible enough to grade free text, but also non-deterministic: run the same evaluation twice and the number can move, and a different judge model gives different numbers. (Ragas also provides deterministic non-LLM and id-based variants of the retrieval metrics when you need repeatability, at the cost of missing paraphrases.) So treat these as directional gauges on a fixed test set, not exact figures to defend to three decimal places. Pin the judge model and version (the same drift problem from the last part), keep a stable evaluation set, and watch the trend rather than the digit. Context recall additionally needs a written reference answer for every test question, which is real labeling work, so one practical approach is to start with the metrics that need no reference and add recall once you have a labeled set.
Four metrics, two per half, and now a wrong answer is diagnosable: precision and recall localize it to retrieval, faithfulness and relevancy to generation. But every metric here quietly assumed the system was allowed to retrieve every document. In the real world it is not: different users are allowed to see different documents, and retrieval has to respect that before it ranks anything. The next part is about RAG with permissions, where the retriever must know who is asking.
Sources: Ragas: list of available metrics.
