English
简体中文繁體中文English日本語DeutschFrançais
Module 04 · Building with LLMs

RAG, letting the model read your material

Build a retrieval-augmented generation system from scratch: chunking documents, vector search, keyword search, query rewriting, reranking, answers with citations and evaluation, and finally teach RepoBot the httpx documentation.

Lessons

  1. 01
    Why RAG

    The same question gets a wrong answer with no material and a right one once the relevant passage of documentation is in the prompt. How RAG works, what it solves, and where it doesn't fit.

    25 minutes · Intermediate
  2. 02
    Splitting documents into chunks

    A hands-on comparison of three chunking methods on the httpx docs: fixed length, by heading, and by heading with a length cap. Along the way we hit a real trap: comments inside code blocks treated as headings.

    35 minutes · Intermediate
  3. 03
    Vector search from scratch

    Write vector search in a few dozen lines of numpy: turn chunks into a matrix of vectors, compute similarities at query time and take the top few. Then evaluate it on 20 questions, find it gets only half right, and see why.

    45 minutes · Intermediate
  4. 04
    Hybrid search and reranking

    Write BM25 keyword search from scratch, fuse it with vector search using RRF, rewrite Chinese questions into English search terms first, and finally add a reranking model. Every step is measured on the same 20 questions.

    50 minutes · Intermediate
  5. 05
    Answers with citations

    Hand the model numbered chunks, require a source for every sentence and an honest "not found" when the material lacks the answer. Then check the citations with code, and see whether it makes things up on a question the docs don't answer.

    35 minutes · Intermediate
  6. 06
    How to tell whether RAG is any good

    Split RAG evaluation into retrieval and answers. Retrieval is measured by hit rate; answers are judged by another model, for correctness against a reference answer and faithfulness against the material. The judge ruled 60 times, I checked every ruling, and found it makes mistakes too.

    45 minutes · Intermediate
  7. 07
    Project: teaching the assistant the project docs

    Put chunking, hybrid search, query rewriting and cited answers into RepoBot to make v2. It gets right the questions v1 got wrong, and solves retrieval for follow-ups like "what about async?" in multi-turn conversation.

    60 minutes · Intermediate

RepoBot v1 got "does httpx follow redirects by default?" wrong, and invented a version history to go with it. This module fixes that: before answering, it looks up the relevant passages in the official httpx documentation, answers from them, and tells the user where the answer came from.

The whole module uses almost no frameworks. Chunking, vector search, BM25, fusion and citation checking are all written by hand, a few dozen lines each. Every retrieval method is evaluated on the same 20 questions, so you'll see exactly how much the numbers move with each addition. Some of the results go against common wisdom: on our data, for example, hybrid search did no better than keyword search alone, and the biggest gain came from one cheap query rewrite.

Why this order

Lesson 1 first proves with a hand-made example that "give it the material and it answers correctly", and explains how RAG works and where it applies. Lessons 2 to 4 follow the pipeline to get the "finding material" step right: first chunking, then retrieval, then better retrieval. Lesson 5 handles the "using material" step: having the model answer with citations, and say honestly when it doesn't know. Lesson 6 evaluates the whole system, not just retrieval but the answers themselves. Lesson 7 puts all of it into RepoBot.

You're done when

  • You can chunk documents by their structure, and spot and fix problems in the chunking (such as code blocks cut in half).
  • Without any framework, you can write vector search and BM25, and fuse them with RRF.
  • You can prepare a set of retrieval evaluation questions for your own documents, compute hit rate and MRR, and choose a retrieval method from them.
  • You can have the model answer only from the retrieved material, cite its sources, and check the citations are valid with code.
  • RepoBot v2 correctly answers the questions v1 got wrong, and gives the documentation source.

Code for this module

Code and program output are shown exactly as they ran, so comments and printed output are in Chinese.