How to tell whether RAG is any good

So far everything we've evaluated is "did retrieval find the right thing". But what the user finally sees is the answer. Retrieval can be entirely right and the answer still wrong: the model may miss something in the material, blend two passages together, or be unable to resist adding its own "knowledge". Conversely, retrieval can miss the most precise chunk and the answer can still be right.

So RAG evaluation has two halves: retrieval, and answers. This lesson covers how to do both, with the focus on evaluating answers and one big trap in doing so.

Evaluate the two halves separately

Evaluating separately means that when something goes wrong you know where to fix it:

Look only at whether the final answer is right and you can't tell the second case from the third, so you're left changing things at random.

The retrieval half is already done: hit rate and MRR from Lesson 3. This lesson evaluates the answers.

What to evaluate in an answer

For RAG answers, the two things most often measured are:

• Correctness: is the answer right? This needs a reference answer to compare against.
• Faithfulness: can every claim in the answer be found in the retrieved material? It doesn't care whether the answer is right, only whether it "went beyond" the material.

Faithfulness is evaluated on its own because it catches a hidden problem: the answer happens to be right, but it's based on the model's memory, not the material. Right this time, it may be wrong next time, and you can't trace it.

Preparing reference answers

I wrote a reference answer for each of the 20 questions from Lesson 3, each based on the passage the retrieval evaluation points to, for example:

{"question": "怎么关闭 SSL 证书校验？", ..., "reference": "传 verify=False，比如 httpx.get(url, verify=False)；用 Client 时在创建客户端时传入。"}
{"question": "httpx 和 requests 在处理重定向上有什么不一样？", ..., "reference": "requests 默认跟随重定向，httpx 默认不跟随；httpx 要显式传 follow_redirects=True（单个请求或 Client 上都可以）。"}

Reference answers don't need to be long; just state clearly the points that must be included. But as you'll see, how you write the reference answer directly affects the scoring.

Letting a model be the judge

With 20 questions and two answers each (one without the docs, one using RAG), judging each by hand is tiring. The common approach is to have another model judge, called an LLM judge (LLM-as-a-judge). The judge is the stronger deepseek-v4-pro, with thinking turned on:

def judge_correct(question, reference, answer):
    return judge(f"""判断"回答"是否正确地回答了"问题"。以"参考答案"为准：回答包含参考答案的要点、且没有和它矛盾的内容，就算正确。
回答比参考答案多说了一些内容没关系，只要多说的部分没有错误。
输出 json：{{"correct": true 或 false, "reason": "一句话理由"}}

问题：{question}
参考答案：{reference}
回答：{answer}""")


def judge_faithful(context, answer):
    return judge(f"""判断"回答"里的每一个事实性说法，是否都能在"资料"里找到依据。
回答说"资料里没有"之类的话不算事实性说法。只要有一处说法在资料里找不到依据，就判为不忠实。
输出 json：{{"faithful": true 或 false, "unsupported": "找不到依据的说法，没有就写空字符串"}}

资料：
{context}

回答：{answer}""")

A few design points:

• The judge isn't the model being judged. A model grading itself tends to overlook its own mistakes.
• Require a reason. Not just true or false, but one sentence on why. As you'll see, the reason is the only clue for spotting the judge's own errors.
• Spell out the criteria. The line "saying more is fine, as long as the extra part isn't wrong" keeps the judge from marking an answer wrong just because it's more detailed than the reference.

The judge function calls the judge using the JSON mode from Module 02, Lesson 4. The full code is in code/04-rag/rag_eval.py; one run makes about 80 calls and costs about $0.1.

The judge's verdicts

不查资料：答对 15/20
RAG：    答对 19/20，忠实于资料 19/20

[不查资料答错] 响应是 404 或 500 时，怎么让它直接抛异常？
    评委：回答中“3xx不会抛”与参考答案“状态码不是2xx时会抛出”矛盾
[不查资料答错] httpx 和 requests 在处理重定向上有什么不一样？
    评委：回答错误地声称httpx默认跟随重定向，与参考答案中“httpx默认不跟随”相矛盾。
[不查资料答错] 怎么限制连接池里最多同时有多少个连接？
    评委：回答遗漏了参考答案中默认 max_connections=100 这一要点，未完整包含参考答案的默认值信息。
[不查资料答错] 怎么显示下载进度？
    评委：回答未使用参考答案中的 response.num_bytes_downloaded 属性，而是手动累计长度
[不查资料答错] 网页返回的中文是乱码，怎么指定解码用的字符集？
    评委：回答未提及参考答案中的 default_encoding 参数指定字符集的方法
[RAG 答错] httpx 和 requests 在处理重定向上有什么不一样？
    评委：回答中关于 requests 暴露的属性 response.next 的描述有误，requests 并没有该属性。
[RAG 不忠实] 怎么显示下载进度？
    找不到依据：显示下载进度需要用流式响应（streaming），并检查 `response.num_bytes_downloaded` 属性

The conclusion looks clear: RAG raised correctness from 15/20 to 19/20. But don't conclude yet. For every "wrong" verdict, I looked at the original answer and checked it against the httpx docs and source.

Checking the judge, one ruling at a time

"3xx doesn't raise": the judge was right. The no-docs answer said raise_for_status() doesn't raise for 3xx. In the httpx source, raise_for_status in _models.py raises HTTPStatusError for anything that isn't 2xx (is_success false), and the error type for 3xx is called "Redirect response". So the answer really was wrong.

Redirects: the no-docs answer was wrong, and the judge was right. Once again it said httpx follows redirects by default, even "by default since 0.20+", exactly the mistake RepoBot v1 made.

Connection pool: the judge was too strict. The no-docs answer correctly said to use httpx.Limits(max_connections=..., max_keepalive_connections=...); it just didn't mention the default of 100. The user asked "how do I limit it", not what the default is. The problem was my reference answer, which casually included "at most 100 connections by default", and the judge treated that as a required point.

Download progress: the judge had a point. The no-docs answer added up downloaded bytes by hand with len(chunk). The httpx docs specifically say that with compression enabled, the length of the decompressed content doesn't match the bytes actually downloaded, so you should use response.num_bytes_downloaded. The answer's approach works without compression, but it isn't the right way.

Encoding: the judge got it wrong. The no-docs answer said to set response.encoding = "gbk" before accessing response.text. I checked the httpx source: Response.encoding has a setter that allows setting the encoding before text is read (setting it afterwards raises ValueError). That's a completely correct alternative. The judge marked it wrong because it "didn't mention default_encoding from the reference answer": it treated the reference answer as the only correct answer.

RAG's redirect answer: the judge got it wrong. The judge said "requests has no response.next attribute". But line 50 of the httpx docs' compatibility.md reads: "The requests library exposes an attribute response.next, which can be used to obtain the next redirect request." The RAG answer followed the docs and was right. The judge didn't look at the material, judged from its own memory, and misremembered.

RAG's download progress: the "unfaithful" verdict was wrong. I reran retrieval for this question; the top two chunks both come from advanced/clients.md and both contain num_bytes_downloaded, and the text says outright to use a streaming response and check this attribute. The answer was entirely faithful to the material; the judge didn't read carefully.

Results after checking

The direction of the conclusion didn't change: RAG is clearly better than answering without the docs, and the no-docs answers made real mistakes on two questions. But the exact numbers changed, and in 60 rulings the judge was wrong 3 times and too strict once.

These 4 problems fall into two kinds:

• The judge ignores the material it was given and judges from its own knowledge. The response.next ruling is one. The judge is an LLM too, and it misremembers too.
• It treats the reference answer as the only answer. The encoding and connection pool rulings are both this. When there are several correct approaches and the reference lists only one, the judge marks the others wrong.

Making the judge more reliable

• Always spot-check. Never take the judge's verdicts as the conclusion directly. At minimum, check every "wrong" by hand, and look at a random sample of "right" ones.
• Require the judge to give reasons. The problems this time were found entirely through the reasons. "requests has no such attribute" looks suspicious at a glance, and one lookup shows the judge was wrong.
• Write reference answers as key points, and allow other correct approaches. Say in the prompt that "other correct approaches beyond the reference answer also count". Put in the reference answer only what the question actually asks.
• The faithfulness judge must look at the material. Stress in the prompt: "judge only from the material, not from your own knowledge".
• Calibrate the judge against human labels. Label a few dozen by hand and see how often the judge agrees with the humans; if agreement is too low, revise the judge's prompt. Module 06, Lesson 2 covers this systematically.

What this lesson concludes

• RAG evaluation splits into retrieval and answers; only by separating them do you know where the problem is.
• Answer evaluation most often looks at correctness and faithfulness.
• An LLM judge saves a great deal of human effort, but it makes mistakes, and makes them confidently. The judge's output is data to be checked, not the final conclusion.

Exercises

1. Edit the reference answers for the connection pool and encoding questions in eval_qa.jsonl to remove what the question didn't ask, add "other correct approaches beyond the reference answer also count" to the judge's prompt, and rerun rag_eval.py. Did the verdicts change?
2. Add "judge only from the material, do not use your own knowledge" to the faithfulness judge's prompt, rerun it, and see whether the verdict on the download progress question changes.
3. Pick 5 questions and act as the judge yourself: decide whether the no-docs answer is correct. Compared with the LLM judge, on how many do you disagree?

Self-check

1. Why should RAG evaluation separate retrieval from answers?

Looking only at the final answer, you can't tell which step failed: the right material wasn't found, or it was found but the model didn't use it well. Evaluating separately tells you whether to fix retrieval (chunking, rewriting, search method) or generation (prompt, model).

2. A RAG answer is correct, but the faithfulness evaluation says it is "unfaithful". What does that mean?

Some of the answer's content has no basis in the retrieved material and probably came from the model's own memory. It happened to be right this time, but for another question it could be wrong, and its source can't be traced. Of course, the judge may also have got it wrong, which needs a human check.

3. The LLM judge marks an answer "wrong" because it "doesn't mention a method from the reference answer". What should you do?

First check whether the method the answer used is also correct. Reference answers often give only one approach, and the judge may mark other correct approaches wrong. If the answer turns out to be right, revise the judge's prompt (allow other correct approaches) or revise the reference answer.