Why RAG

In the last lesson, RepoBot v1 got "does httpx follow redirects by default?" wrong, and invented a version history to cover it. Changing the prompt or turning on thinking can only reduce this kind of error, not eliminate it, because the model can only answer from memory, and in its memory httpx and requests are mixed together.

What does a person do with a question like this? Look it up in the documentation. In this lesson we have the model "look it up" too.

Experiment: give it a passage of documentation

The httpx docs have a page, compatibility.md, devoted to how it differs from requests, and one of its sections is called "Redirects". I first find that section with the chunking function covered in Lesson 2, then ask the same question twice: once with no material, and once with that section in the prompt.

section = next(c for c in split_by_heading(load_docs()["compatibility.md"]) if c.startswith("## Redirects"))

answer, tokens = ask([{"role": "user", "content": QUESTION + "用两三句话回答。"}])

prompt = f"""根据下面的 httpx 文档片段回答问题。文档里没有提到的，就说文档里没有。

<doc>
{section}
</doc>

问题：{QUESTION}用两三句话回答。"""
answer, tokens = ask([{"role": "user", "content": prompt}])

The passage it found:

## Redirects

Unlike `requests`, HTTPX does **not follow redirects by default**.

We differ in behaviour here [because auto-redirects can easily mask unnecessary network
calls being made](https://github.com/encode/httpx/discussions/1785).

You can still enable behaviour to automatically follow redirects, but you need to
do so explicitly...

```python
response = client.get(url, follow_redirects=True)
```

Or else instantiate a client, with redirect following enabled by default...

```python
client = httpx.Client(follow_redirects=True)
```

The two answers (full code in code/04-rag/why_rag.py; your wording will differ):

== 不给资料（输入 19 词元）：
会，httpx 默认会自动跟随重定向（最多 20 次）。可以在请求中用 `follow_redirects=False` 关闭，或用 `max_redirects` 调整次数。

== 给了文档片段（输入 169 词元）：
不会。文档明确说明 httpx 与 `requests` 不同，**默认不跟随重定向**。如果想自动跟随，必须显式设置，例如在请求中用 `follow_redirects=True`，或在创建客户端时用 `httpx.Client(follow_redirects=True)`。

With no material it's wrong, the same mistake RepoBot v1 made. Given 150 tokens of documentation, it's immediately right, and it also explains why and how to turn redirects on. The extra cost is 150 input tokens, about $0.00005 at the prices from Module 01, Lesson 4.

That is the whole idea of RAG: first find the relevant material, then have the model answer from it.

What RAG is

RAG stands for Retrieval-Augmented Generation. The name is long, but it breaks down into three steps: retrieval, augmentation (adding what was found to the prompt), and generation.

In the experiment above, "finding the Redirects section" was done by hand; I already knew where the answer was. A real RAG system has to do this step automatically: whatever the user asks, the program finds the few most relevant passages in a large pile of documents. The full flow has two parts:

提前准备（只做一次，文档更新时再做）：

  文档 ──▶ 切成小块 ──▶ 为每块建立索引（向量、关键词）──▶ 存起来
          （第 2 课）     （第 3、4 课）

每次提问时：

  用户问题 ──▶ 检索：找出最相关的几块 ──▶ 把这几块和问题一起放进提示词 ──▶ 模型回答
                （第 3、4 课）              （第 5 课：要求它注明引用）

The rest of this module builds each box in that diagram: Lesson 2 chunking, Lesson 3 vector search, Lesson 4 keyword search and fusing the two, Lesson 5 having the model answer with citations, Lesson 6 evaluating the whole system, and Lesson 7 putting it all into RepoBot.

What RAG solves

Things the model doesn't know. Your company's internal docs, your project's code, the release notes for last week's new version: the model never saw them in training. RAG hands them to it on the spot.

Things the model remembers wrong. The httpx redirect example is one. The model "knows" httpx but has it mixed up. With the original text in front of it, it doesn't have to rely on memory.

Traceability. The program knows which passages an answer was based on and can show them to the user. The user can open the source and check, and when something is wrong you can tell whether retrieval failed or the model misread.

Cheap updates. When the docs change, you reprocess only the pages that changed. By comparison, "teaching" new knowledge to a model through training costs far more and works unreliably (Module 10, Lesson 1 explains why fine-tuning is a poor way to add knowledge).

Why not put all the documents in

Module 01, Lesson 4 ran this experiment: the whole httpx documentation is about 29,000 tokens, and with all of it in the prompt the model also answers correctly. So why bother with retrieval?

Because stuffing in 29,000 tokens every time costs dozens of times more than the few hundred that are relevant, and is much slower, while each question only needs a tiny part of it. The httpx docs are small; your company's knowledge base might be tens of millions of tokens and simply won't fit in the context. And the more irrelevant content you include, the more likely the model is to be distracted and look in the wrong place.

So each approach has its range:

Consider first whether everything fits. If it does, do that, and don't rush into RAG.

Where RAG doesn't fit

• The question needs all of the material. "Summarise this document" or "how many parameters does the document mention in total" need the full text; a few retrieved passages aren't enough.
• The material itself is poor. If the docs are outdated, contradictory or vague, RAG just makes the model answer from wrong material, and because there is a "source", the answer looks more trustworthy.
• The answer isn't in any document. Say some httpx behaviour exists only in the source code and the docs don't mention it. Then the model needs to read the source, which is what the agent in Module 05 does.
• It needs reasoning, not lookup. "Why does my code throw this error?" Finding documentation isn't enough; the model has to understand the user's code. RAG can supply relevant docs as support, but it mostly comes down to the model's own ability.

Exercises

1. Run code/04-rag/why_rag.py and look at the two answers you get.
2. Try another question, such as "does httpx's Response have an ok attribute?". Find the relevant section in compatibility.md by hand (hint: search for "Checking for success") and modify why_rag.py to make the same comparison.
3. Replace the documentation in why_rag.py's prompt with something unrelated (say, a section of timeouts.md) and ask the redirect question again. How does the model answer? Does it say "the documentation doesn't mention this"?

Self-check

1. What do the three letters of RAG stand for, and what does each step do?

Retrieval: find the material most relevant to the question. Augmented: add what was found to the prompt. Generation: the model writes the answer from the material.

2. Your material is only 20,000 tokens and gets asked about a few dozen times a day. Would you use RAG?

Usually not. When the material is small and calls are infrequent, putting all of it in the prompt is simpler and more reliable, and fixed material at the start can hit the cache, so the cost stays low. RAG suits material too large to fit in the context, or calls so frequent that including everything is too expensive.

3. Why can RAG make things worse when the material is poor?

The model answers from whatever was retrieved. If the material is wrong, the answer is wrong, and because it comes with a "source", users are more likely to believe it. RAG is only as good as the material behind it.