Answers with citations

With retrieval working, the material it finds has to be handed to the model. How you hand it over, and what you ask of the model once it has it, decides whether the final answer can be trusted.

This lesson has three goals: the model answers only from the material, marks which passage each sentence is based on, and honestly says so when the material doesn't contain the answer. Then we write a program to check automatically whether it did.

Numbering the material

Retrieval uses last lesson's best-performing "rewrite + hybrid RRF", taking the top 5 chunks. Each chunk is wrapped in a tag carrying its number and source:

def build_context(results):
    parts = []
    for i, (_, file, text) in enumerate(results, 1):
        parts.append(f'<doc id="{i}" source="{file}">\n{text}\n</doc>')
    return "<docs>\n" + "\n".join(parts) + "\n</docs>"

In the prompt it looks like this (content omitted):

<docs>
<doc id="1" source="compatibility.md">
## Redirects
Unlike `requests`, HTTPX does **not follow redirects by default**. ……
</doc>
<doc id="2" source="compatibility.md">
……
</doc>
……
</docs>

问题：httpx 默认会自动跟随重定向吗？怎么开启？

The number is for the model to cite; source is for the model's reference and makes it easy for the program to show the user at the end. Module 02, Lesson 1 covered using tags to separate material from instructions; here the <doc> tag has one more job: it clearly marks the boundaries of each passage, so the model doesn't merge two passages into one.

The prompt

SYSTEM = """你是 httpx 的答疑助手。只根据 <docs> 里的文档片段回答用户的问题。

规则：
- 每句话后面用 [编号] 注明依据的是哪个片段，可以同时引用多个，如 [1][3]。
- 片段里没有的信息不要写，哪怕你自己知道。
- 如果这些片段不足以回答问题，就直接说"文档里没有找到相关说明"，不要猜。
- 用中文回答，简洁，代码保持原样。"""

The phrase "even if you know it yourself" in the second rule matters. The model already knows a lot about httpx, but what it knows may be wrong, such as redirects being followed by default. Requiring it to use only the material replaces the correctness of the model's memory with the correctness of the material. The price: things the material happens not to mention, which the model does in fact know, can't be said either. Lesson 7 runs into such a case.

The third rule gives the model a clear "way out". Without it, when the material falls short the model tends to fill the gap with its own knowledge, which is exactly what we want to avoid.

Results

I prepared 4 questions: two with clear answers in the docs; one the docs don't mention at all (HTTP/3; I searched all of the httpx docs and it never appears); and one whose answer requires reading the material carefully (does httpx have built-in retries?). Full code in code/04-rag/citations.py; the output:

问：httpx 默认会自动跟随重定向吗？怎么开启？
检索到：[1] compatibility.md  [2] compatibility.md  [3] quickstart.md  [4] http2.md  [5] compatibility.md
答：httpx 默认**不会**自动跟随重定向，所有 HTTP 方法都是如此 [1][3]。

你可以在单个请求上显式开启 [1][3]：

```python
response = client.get(url, follow_redirects=True)
```

或者在创建 `Client` 时就默认开启 [1]：

```python
client = httpx.Client(follow_redirects=True)
```

开启后，可以通过 `response.history` 属性查看被跟随的重定向响应列表，按发生顺序排列 [3]。
检查：引用了 [1, 3]，不存在的编号 无，没有引用的句子 0 句（输入 1002 词元）

问：httpx 支持 HTTP/3 吗？
检索到：[1] async.md  [2] http2.md  [3] advanced/transports.md  [4] http2.md  [5] advanced/transports.md
答：文档里没有找到相关说明。
检查：引用了 []，不存在的编号 无，没有引用的句子 1 句（输入 1128 词元）
    未引用：文档里没有找到相关说明。

问：httpx 自带请求失败自动重试的功能吗？
检索到：[1] third_party_packages.md  [2] advanced/transports.md  [3] compatibility.md  [4] async.md  [5] advanced/transports.md
答：httpx 自带连接层面的重试功能，通过直接实例化 transport 来使用：当发生 `httpx.ConnectError` 或 `httpx.ConnectTimeout` 时，请求会按给定的次数重试，从而在链路不稳定的网络下更平滑地运行 [2]。
（代码示例省略）
如果只需要专门的 HTTPX 重试层，可以看 httpx-retries，它是一个 for HTTPX 的 retry layer [1]。若需要其他形式的重试行为（例如处理读/写错误，或针对 `503 Service Unavailable` 做出反应），文档建议考虑 tenacity 这类通用工具 [2]。
检查：引用了 [1, 2, 4]，不存在的编号 无，没有引用的句子 0 句（输入 908 词元）

(The result for the question about disabling SSL verification was also correct and is omitted here.)

Redirects: correct, and it cited two sources, since both compatibility.md and quickstart.md cover this.

HTTP/3: not in the docs, so the model said so and not a word more. Compare Module 01, Lesson 2, where the same model, with no material and no rules, invented a nonexistent CVE number.

Retries: this is the most telling question. "Does httpx have built-in retries?" is easy to answer from memory with a flat "yes" or "no". Working from the material, it gave an accurate, carefully qualified answer: the only built-in retries are for connection failures, and read/write errors and 503s need other tools. That is exactly what the original httpx docs say; I checked the same point in the docs back in Module 01, Lesson 2.

Checking citations with code

The model says it cited [2], but does [2] actually exist? Does every sentence have a citation? These can be checked with code:

def check_citations(text, n_docs):
    """找出回答里所有的 [数字]，检查它们是否都在 1..n_docs 范围内，以及有多少句话没有引用。"""
    cited = [int(x) for x in re.findall(r"\[(\d+)\]", text)]
    invalid = sorted({c for c in cited if not 1 <= c <= n_docs})
    prose = re.sub(r"```.*?```", "", text, flags=re.S)  # 代码块是照抄文档的，不要求逐行引用
    sentences = [s.strip() for s in re.split(r"(?<=[。！？])|\n+", prose) if len(s.strip()) > 8]
    # 以冒号结尾的句子是在引出下面的代码，引用通常写在代码块后面，也不算
    uncited = [s for s in sentences if not re.search(r"\[\d+\]", s) and not s.endswith(("：", ":"))]
    return sorted(set(cited)), invalid, uncited

It does two things:

• Whether cited numbers exist. If there are only 5 passages and the model cites [7], it is inventing a source. This didn't happen once in the 4 questions, but it does happen when there is a lot of material and the answer is long.
• Which sentences have no citation. A sentence without a citation may be something the model added on its own.

I revised this checker once. The first version treated every line of code in a code block as a "sentence without a citation", so every question with code produced a pile of false alarms. Code is copied from the docs and doesn't need line-by-line citations, so now it strips code blocks before checking. The only "uncited" sentence left is the "no relevant information found in the documentation" line, which needs no citation anyway.

Checking with code has an obvious limit: it can check the format, not the content. The model cited [2], but whether [2] actually supports the sentence is something the program can't judge. That needs a person, or another model to judge it, which is the "faithfulness" evaluation of the next lesson.

Showing citations to the user

To users, numbers like [1] and [3] mean nothing; they need to be replaced by real sources. RepoBot v2 does this: after the answer ends, it finds which numbers the answer actually cited and lists the matching documents:

来源：[1] compatibility.md  [2] quickstart.md

On a web page, you can turn the numbers into links that jump straight to the right place in the docs. Users can check for themselves, and when something is wrong it's obvious which passage misled the model.

Problems you may run into

The model doesn't cite, or cites only once at the end. Give an example in the prompt showing the citation format you expect.

It cites, but cites wrongly. The answer says something from passage A but marks it [B]. This is likely when passages are similar in content. You can ask the model to quote the key sentence of the source when citing, to make checking easier, at the cost of longer answers.

The passages contradict each other. For example, one is from an old version of the docs and one from a new version. The model may pick either one at random, or blend them. The fix is on the retrieval side: tag chunks with version information and retrieve only the current version's docs.

Too conservative. The material does contain the answer, just stated indirectly, but the model replies "not in the documentation". This is a side effect of requiring "only from the material". You can say in the prompt that "things mentioned indirectly in the material may be answered, but say that it's an inference".

Exercises

1. Add a question to citations.py: "Which performs better, httpx or requests?" The docs contain no direct performance comparison; see how the model answers.
2. Delete the rule "don't write information that isn't in the passages, even if you know it yourself" from the prompt and ask the HTTP/3 question again. Does the model answer from its own knowledge, and is what it says correct?
3. Add a check to check_citations: if the answer cites a passage, that passage should contain at least one of the code identifiers in the answer (such as follow_redirects). This is a rough "content check"; think about when it would misjudge.

Self-check

1. Why does the prompt say "even if you know it yourself, don't write what isn't in the passages"?

The model's own knowledge may be wrong; for example, it misremembers httpx's default redirect behaviour. Requiring answers only from the material replaces the correctness of the model's memory with the correctness of the material, and makes every sentence traceable. The price is that information the material happens not to contain can't be given.

2. If the program check passes (every cited number exists, every sentence has a citation), does that mean the answer is reliable?

No. The program can only check the format: whether numbers exist and whether citations are marked. It can't judge whether the cited passage really supports the sentence. Faithfulness of content needs a human check, or another model to judge it.

3. Why give the model an explicit phrase like "no relevant information found in the documentation"?

Without a clear way out, when the material falls short the model tends to fill in with its own knowledge, which is exactly where errors come in. With a fixed phrase, it knows "I don't know" is an acceptable answer, and the program can use that phrase to detect cases where retrieval didn't find the answer.