Project: teaching the assistant the project docs

This lesson puts everything from Module 04 into RepoBot to make the second version. You'll meet a problem that never came up in single questions: in a multi-turn conversation, the user's follow-up often finds nothing when searched on its own.

What done looks like

• Ask "does httpx follow redirects automatically by default?" and the answer is "no", with compatibility.md listed as the source.
• Every answer marks its sources with [numbers] and ends with the documents actually cited.
• Ask about something not in the docs (such as HTTP/3) and it answers "no relevant information found in the documentation" without inventing anything.
• Ask "how do I set a timeout in httpx?" followed by "what about the async client?", and the second question retrieves the timeout docs.
• On the second launch, the document vectors aren't recomputed.
• eval_retrieval.py reaches a 100% top-5 hit rate on the 20 evaluation questions.

Structure

The code is in projects/repobot/v2/, with two more files than v1:

llm.py             客户端、计费、带重试的请求（从 v1 的 repobot.py 里挪出来）
retrieval.py       切分、BM25、向量检索、RRF、重排、查询改写（04 模块第 2～4 课）
repobot.py         对话程序：在 v1 的基础上加上检索和引用
eval_retrieval.py  用 20 道题评估检索效果（04 模块第 3 课）

The code in retrieval.py was covered piece by piece in earlier lessons, so here we only discuss three new problems that come up when assembling it.

Problem 1: follow-ups find nothing

In Module 04's exercise code, every question stood alone. In a conversation, though, users ask like this:

你：怎么给 httpx 设置 10 秒的超时？
你：那异步客户端呢？

Search with the second question as-is, and "what about the async client?" finds the docs about async but not the ones about timeouts, because the word "timeout" isn't in the question at all.

v2's fix is to include the recent conversation in the query rewriting step, so the model first works out what the user is actually asking, then produces the search terms:

REWRITE_PROMPT = """你要为一个 httpx 答疑助手生成文档检索词。httpx 的文档是英文的。

根据"最近的对话"理解用户"最新的问题"到底在问什么（比如"那异步呢"要结合上文补全），
然后输出一行英文检索词：包含问题的完整英文表述，以及文档里可能出现的参数名、类名、术语。只输出这一行。"""


class QueryRewriter:
    def rewrite(self, question, history=()):
        recent = "\n".join(f"{m['role']}: {m['content'][:300]}" for m in list(history)[-4:])
        ……
            text, self.last_usage = llm.chat([
                {"role": "system", "content": REWRITE_PROMPT},
                {"role": "user", "content": f"最近的对话：\n{recent or '（无）'}\n\n最新的问题：{question}"},
            ])

It takes only the last 4 messages, each up to 300 characters: enough to resolve references without making the rewrite step too expensive. You can see the effect in the output below.

Problem 2: what goes into the history

Every turn puts the 5 retrieved passages into the prompt, about a thousand tokens. If those were also stored in the conversation history, after ten turns the history would hold over ten thousand tokens of old documents, which is both expensive and distracting for the model.

v2 puts the documents only into this turn's user message; the history keeps only the user's original question and the model's answer:

            messages = ([{"role": "system", "content": SYSTEM}] + history +
                        [{"role": "user", "content": build_context(results) + f"\n\n问题：{question}"}])
            ……
        history += [{"role": "user", "content": question}, {"role": "assistant", "content": text}]

The model's answer already contains the key points it took from the docs, so when later turns need them, they can be found in the answer.

Problem 3: don't recompute the vectors on every launch

Computing the 196 chunks with the embedding model on every launch takes over ten seconds. If the docs don't change, the result is the same every time, so it can be cached:

        key = hashlib.sha256((EMBED_MODEL + json.dumps(self.chunks)).encode()).hexdigest()[:16]
        path = cache_dir / f"vectors-{key}.npy"
        if path.exists():
            self.matrix = np.load(path)
        else:
            self.matrix = self.embedder.encode(["passage: " + t for t in texts], normalize_embeddings=True, batch_size=32)
            np.save(path, self.matrix)

The cache file is named from a hash of "embedding model name + the content of every chunk". Change a single character of the docs, or switch embedding models, and the hash changes, so it recomputes and never reuses stale vectors. Query rewrites are cached the same way in .cache/rewrites.json.

Retrieval results

cd projects/repobot/v2
pip install -r requirements.txt
export HF_ENDPOINT=https://hf-mirror.com
python eval_retrieval.py
python eval_retrieval.py --rerank

== 不加重排
前 1 名命中：80%
前 3 名命中：100%
前 5 名命中：100%
MRR：0.892
== 加重排
前 1 名命中：85%
前 3 名命中：95%
前 5 名命中：100%
MRR：0.902

Without reranking the MRR is 0.892, well above the 0.772 of Lesson 4's "rewrite + hybrid RRF". The search algorithm is exactly the same; the only difference is that the rewrite prompt became the new version above: "output one line of English search terms: a complete English phrasing of the question, plus…". Lesson 4's version said "an English translation of the question, plus…", and had no conversation context.

This shows again how much the rewrite prompt affects retrieval. It's also why you rerun the evaluation after every change: you think you've just tweaked one sentence of a prompt, and the metrics may move.

On this version, reranking only lifts MRR from 0.892 to 0.902, and the top-3 hit rate actually falls from 100% to 95%. An extra second buys very little, so v2 leaves reranking off by default; add --rerank when you want it.

One conversation

printf '%s\n' "httpx 默认会自动跟随重定向吗？" "那默认最多跟随几次？" "怎么给 httpx 设置 10 秒的超时？" \
  "那异步客户端呢？" "httpx 支持 HTTP/3 吗？" "今天北京天气怎么样？" | python repobot.py --show-query

--show-query shows each turn's search terms and retrieved documents. My run's output (with a few code examples removed, otherwise unchanged):

你：httpx 默认会自动跟随重定向吗？
[检索词] Does httpx follow redirects by default? follow_redirects parameter, redirect, httpx.Client, AsyncClient, max_redirects, history, redirection behavior
[检索到] [1] compatibility.md  [2] quickstart.md  [3] compatibility.md  [4] compatibility.md  [5] http2.md
RepoBot：不会。与 `requests` 不同，HTTPX 默认**不跟随重定向**[1]，对所有 HTTP 方法默认都不跟随重定向[2]。
（启用方法和示例代码省略）
来源：[1] compatibility.md  [2] quickstart.md
[回答输入 1047（缓存命中 0），输出 180，本轮 0.00060 美元（含改写），累计 0.00060 美元]

你：那默认最多跟随几次？
[检索词] httpx follow_redirects max_redirects default maximum number of redirects
[检索到] [1] quickstart.md  [2] compatibility.md  [3] advanced/resource-limits.md  [4] compatibility.md  [5] advanced/transports.md
RepoBot：文档里没有找到相关说明。检索到的片段只说明了 HTTPX 默认不跟随重定向，以及如何通过 `follow_redirects=True` 启用[1][2]，但没有提到最大跟随次数的默认值。

建议查阅 HTTPX 官方文档中关于重定向或多重定向（`max_redirects`）的页面确认。
来源：[1] quickstart.md  [2] compatibility.md

你：怎么给 httpx 设置 10 秒的超时？
[检索词] httpx set timeout 10 seconds how to configure timeout Timeout timeout parameter httpx.Client(timeout=...) httpx.get(timeout=TimeOut(...)) DEFAULT_TIMEOUT_CONFIG connect read write pool timeout
[检索到] [1] advanced/extensions.md  [2] advanced/timeouts.md  [3] advanced/timeouts.md  [4] advanced/timeouts.md  [5] advanced/timeouts.md
RepoBot：可以给客户端实例设置默认超时，例如设置 10 秒对所有请求生效：
（代码示例省略）
来源：[2] advanced/timeouts.md  [4] advanced/timeouts.md

你：那异步客户端呢？
[检索词] asynchronous client timeout configuration httpx.AsyncClient(timeout=10.0) default timeout per-request async
[检索到] [1] advanced/timeouts.md  [2] advanced/timeouts.md  [3] advanced/timeouts.md  [4] compatibility.md  [5] quickstart.md
RepoBot：异步客户端同样支持超时设置，用法与同步客户端一致。
（代码示例省略）
不过需要说明：文档片段中给出的示例都写作 `httpx.Client(...)`，没有出现 `httpx.AsyncClient(...)` 的写法，因此片段里没有专门针对异步客户端的示例，但相同参数适用于客户端实例的默认超时与单请求超时设置[1][3]。
来源：[1] advanced/timeouts.md  [2] advanced/timeouts.md  [3] advanced/timeouts.md

你：httpx 支持 HTTP/3 吗？
[检索词] httpx HTTP/3 support HTTP/3 QUIC protocol
[检索到] [1] index.md  [2] advanced/proxies.md  [3] index.md  [4] index.md  [5] http2.md
RepoBot：文档里没有找到相关说明。

检索到的片段只提到 HTTPX 支持 HTTP/1.1 和 HTTP/2[3][4]，并且 HTTP/2 默认不启用、需要安装可选依赖并显式开启[5]，但没有提到 HTTP/3。

建议查阅 HTTPX 官方文档确认是否支持 HTTP/3。
来源：[3] index.md  [4] index.md  [5] http2.md

你：今天北京天气怎么样？
[检索词] Beijing weather forecast today current temperature conditions
[检索到] [1] troubleshooting.md  [2] logging.md  [3] troubleshooting.md  [4] exceptions.md  [5] troubleshooting.md
RepoBot：抱歉，我只负责 httpx 相关的答疑，不回答天气这类无关问题。如果你有 httpx 的使用问题，欢迎继续问我。
[回答输入 2481（缓存命中 640），输出 29，本轮 0.00069 美元（含改写），累计 0.00388 美元]

Turn by turn

Redirects: correct. v1 got this wrong and invented a version history. v2 says "no" and gives two sources.

The follow-ups were understood correctly. "Then what's the default maximum number of redirects?" was rewritten to httpx follow_redirects max_redirects default maximum number of redirects; "what about the async client?" was rewritten to asynchronous client timeout configuration httpx.AsyncClient(timeout=10.0), and everything retrieved was from timeouts.md. Without conversation context in the rewrite, the second question would only find the docs about async.

"Default maximum number of redirects": it says the docs don't have it. That answer is right: the httpx docs really don't give this default; it exists only in the source (in Module 03, Lesson 5 we found it in httpx/_config.py: 20). Interestingly, v1 got this one right from memory. v2, strictly required to "answer only from the material", can't answer it. That's the price mentioned in Lesson 5: what the material doesn't say can't be said, even if the model knows it.

Async client: a carefully measured answer. All the examples in the docs use httpx.Client; it says so plainly while pointing out that the same parameter applies. That's exactly what we want: not pretending the docs contain something they don't.

HTTP/3: nothing invented.

Weather: declined, but money wasted. It still did a rewrite and a search, and stuffed 5 completely irrelevant passages into the prompt. It would be better to decide before retrieval whether the question has anything to do with httpx and decline immediately if not. The "guardrails" in Module 06, Lesson 5 do this.

Cost. About $0.0006 to $0.0007 per turn, including the rewrite call, somewhat more than v1's $0.0001 to $0.0005 per turn, mainly because of the roughly thousand tokens of docs. Cache hits keep growing because the system prompt and conversation history form a fixed prefix; only the docs in the last user message change each time.

Where v2 falls short

• If the docs don't say it, it can't answer. The default number of redirects, the internal logic of Limits, exactly when a certain exception is raised: these answers are in the source.
• It searches only once. If the first search misses, it gets no chance to try again with different terms.
• It searches first no matter what the question is. Even a question about the weather.

The first two need RepoBot to decide for itself to "look up the docs" or "read the source", and choose its next step based on what it finds. That's the agent in Module 05.

Questions this project should answer

• Why this design? Query rewriting + BM25 + vector RRF is the combination that scored best on the 20 evaluation questions without needing extra compute. Reranking helps very little, so it's optional.
• Where will it fail? Questions whose answers are only in the source, not the docs; questions that need several documents combined; questions where the first search misses.
• How is it evaluated? Retrieval with eval_retrieval.py, answer quality with the LLM judge from Lesson 6.
• What do you look at when something goes wrong? Turn on --show-query: first check the search terms, then the retrieved documents, and finally whether the model used the documents correctly. Most problems are in the first two steps.
• Can it be cheaper? It can decide before retrieval whether the question relates to httpx and decline unrelated ones directly, saving the rewrite, the search and over a thousand input tokens.
• Does it need an agent? Not as of this version: the flow is fixed, rewrite, search, answer. Only when it needs "if the search finds nothing, try another way" does it need one.

Exercises

1. Test v2 with the 5 questions you wrote in the exercise for v1 (Module 03, Lesson 5). Is there a question v1 got right that v2 can't answer?
2. Remove the history parameter from QueryRewriter.rewrite (always pass an empty history), ask "how do I set a timeout?" and "what about the async client?" again, and see what the second question retrieves.
3. In repobot.py, call the model once before retrieval to decide "is this question related to httpx?", and if not, reply with the refusal directly without searching. Compare the cost of the weather turn before and after.

Self-check

1. If the user asks "what about the async client?", what goes wrong when you search with that sentence as-is? How does v2 fix it?

The sentence contains nothing like "timeout", so the search only finds general docs about async, not the content on timeout settings for the async client. v2 includes the last few turns of conversation when rewriting the query, so the model first understands that the user is really asking "how do I set a timeout on the async client", then produces complete search terms.

2. Why do the retrieved documents go only into the current turn's message and not into the conversation history?

Each turn's documents are about a thousand tokens. Storing them all in the history would make it grow with every turn, which is expensive and distracting for the model. The model's answer already contains the key points taken from the docs, so storing the answer is enough.

3. v1 answered "the default maximum number of redirects" correctly from memory, while v2 says the docs don't have it. Does that make v2 worse than v1?

No. v2 answers only from the docs, and the docs really don't contain this, so it honestly says it found nothing. It trades "may not be able to answer" for "whatever it answers can be traced to a source". v1 happened to be right this time, but it will confidently get other questions wrong. The fix for v2 is to let it look up more material, such as the source code, not to relax the "only from the material" requirement.