Vector search from scratch

Module 01, Lesson 5 introduced embeddings: text with similar meaning has similar vectors. The last lesson split the httpx docs into 196 chunks. Put the two together and you have a semantic search engine: compute a vector for every chunk in advance, compute the question's vector when the user asks, and find the chunks most similar to it.

Many tutorials have you install a vector database straight away. This lesson doesn't. We write one ourselves in numpy, a few dozen lines in total. Once it's written, you'll see that the core of vector search is a single matrix multiplication. Then we evaluate it on 20 questions to see whether it's actually any good.

Building the index

import numpy as np
from sentence_transformers import SentenceTransformer


class VectorIndex:
    def __init__(self, model_name):
        self.model = SentenceTransformer(model_name)
        # e5 系列要求给查询和文档分别加上前缀，这是它训练时的约定
        self.q_prefix, self.d_prefix = ("query: ", "passage: ") if "e5" in model_name else ("", "")
        self.chunks = []  # [(文件名, 文本), ...]
        self.matrix = None  # 每一行是一个块的向量

    def build(self, chunks):
        self.chunks = chunks
        texts = [self.d_prefix + text for _, text in chunks]
        self.matrix = self.model.encode(texts, normalize_embeddings=True, batch_size=32)

build hands all the chunks to the embedding model at once and gets back a matrix: 196 chunks, one 512-dimensional vector each, so a matrix of 196 rows and 512 columns. normalize_embeddings=True scales every vector to length 1, so that computing similarity later only takes a dot product.

Each chunk also records which file it came from. We'll use that to judge whether retrieval was right, and in Lesson 5 to tell the user where an answer came from.

Searching

    def search(self, query, k=5):
        q = self.model.encode([self.q_prefix + query], normalize_embeddings=True)[0]
        scores = self.matrix @ q  # 向量都归一化过了，点积就是余弦相似度
        top = np.argsort(-scores)[:k]
        return [(float(scores[i]), *self.chunks[i]) for i in top]

self.matrix @ q multiplies the matrix by a vector: each of the 196 rows takes a dot product with the question vector, computing the similarity between the question and every chunk in one go. np.argsort(-scores) sorts by similarity from high to low, and we take the first k.

That's a complete vector search. Let's try it:

BAAI/bge-small-zh-v1.5：196 个块，向量 (196, 512)，建索引用了 15.7 秒

示例：「怎么关闭 SSL 证书校验？」最相似的块来自 advanced/ssl.md，相似度 0.628
### Enabling and disabling verification

By default httpx will verify HTTPS connections, and raise an error for invalid SSL cases...

A question in Chinese found the section of the English docs about certificate verification. Building the index took 15.7 seconds, mostly spent loading the model for the first time and computing 196 vectors. The search itself takes almost no time.

How do you know whether it's good

One question answered correctly proves nothing. To evaluate systematically, you need a set of questions with known answers.

I prepared 20 questions in Chinese in code/04-rag/eval_qa.jsonl. Each is labelled with the file the answer should be in and a keyword: a retrieved chunk counts as correct only if it comes from that file and contains that keyword.

{"question": "怎么把超时完全关掉，让请求一直等下去？", "file": "advanced/timeouts.md", "keyword": "timeout=None"}
{"question": "服务器要求 Digest 认证怎么办？", "file": "advanced/authentication.md", "keyword": "DigestAuth"}
{"question": "有些域名不想走代理，环境变量怎么设置？", "file": "environment_variables.md", "keyword": "NO_PROXY"}
……

The judgement uses "file + keyword" rather than chunk numbers because change the chunking method and every chunk number changes, while files and keywords stay the same. That way the same question set can evaluate a different chunking method directly. I confirmed with a script that every keyword really does appear in its file.

Then we compute a few metrics:

• Top-1 hit rate: the proportion of questions where the first-ranked chunk is correct.
• Top-3 and top-5 hit rate: the proportion where a correct chunk is among the first few. RAG usually hands the model several chunks at once, so this metric matters more.
• MRR (mean reciprocal rank): a correct chunk ranked 1st scores 1, 2nd scores 1/2, 3rd scores 1/3, not found scores 0, averaged over all questions. It captures both "was it found" and "how high was it ranked".

def evaluate(search, questions, k=5):
    """返回第 1 名命中率、前 3 名命中率、前 5 名命中率、MRR，以及没找到的题。"""
    ranks, misses = [], []
    for qa in questions:
        results = search(qa["question"], k)
        rank = next((i + 1 for i, r in enumerate(results) if is_hit(r, qa)), None)
        ranks.append(rank)
        if rank is None:
            misses.append((qa, results[0]))
    n = len(questions)
    hit = lambda top: sum(1 for r in ranks if r and r <= top) / n
    mrr = sum(1 / r for r in ranks if r) / n
    return hit(1), hit(3), hit(5), mrr, misses

evaluate takes a search function, not a particular index. Next lesson's keyword search and hybrid search can all be evaluated with it, and their results compared directly.

The result: only half

20 道题：第 1 名命中 20%，前 3 名命中 40%，前 5 名命中 50%，MRR 0.303
  没找到：怎么知道一个响应实际用的是 HTTP/1.1 还是 HTTP/2？（应在 http2.md）→ 第 1 名是 advanced/clients.md：!!! hint
  没找到：服务器要求 Digest 认证怎么办？（应在 advanced/authentication.md）→ 第 1 名是 advanced/ssl.md：### Enabling and disabling verification
  没找到：怎么让请求走 HTTP 代理？（应在 advanced/proxies.md）→ 第 1 名是 advanced/clients.md：!!! hint
  没找到：下载很大的文件时，怎么一块一块地读，而不是一次读进内存？（应在 quickstart.md）→ 第 1 名是 advanced/clients.md：## Multipart file encoding
  没找到：响应是 404 或 500 时，怎么让它直接抛异常？（应在 quickstart.md）→ 第 1 名是 advanced/timeouts.md：HTTPX is careful to enforce timeouts eve
  没找到：异步发请求应该用哪个类？（应在 async.md）→ 第 1 名是 async.md：# Async Support
  没找到：httpx 和 requests 在处理重定向上有什么不一样？（应在 compatibility.md）→ 第 1 名是 advanced/clients.md：!!! hint
  没找到：写测试时，怎么不真的发网络请求，而是返回一个假的响应？（应在 advanced/transports.md）→ 第 1 名是 advanced/clients.md：!!! hint
  没找到：想在每个请求发出之前和收到响应之后都执行一段代码，比如打日志，怎么做？（应在 advanced/event-hooks.md）→ 第 1 名是 advanced/timeouts.md：HTTPX is careful to enforce timeouts eve
  没找到：有些域名不想走代理，环境变量怎么设置？（应在 environment_variables.md）→ 第 1 名是 advanced/clients.md：!!! hint

Of 20 questions, only half have a correct chunk in the top 5. Put that into RAG, and for half the questions the model gets irrelevant material.

Let's try a different embedding model. intfloat/multilingual-e5-small is an embedding model trained specifically for many languages:

python code/04-rag/vector_search.py intfloat/multilingual-e5-small

20 道题：第 1 名命中 30%，前 3 名命中 50%，前 5 名命中 65%，MRR 0.418

Somewhat better: the top-5 hit rate goes from 50% to 65%, but it's still not good.

What's going wrong

Crossing languages. The questions are in Chinese and the docs are in English. bge-small-zh-v1.5 is trained mainly for Chinese and has limited understanding of English. multilingual-e5-small is trained for many languages, so it does better. But mapping Chinese questions and English documents into the same vector space is inherently harder than working within one language.

The "catch-all chunk". Look closely at the questions it missed, and the same chunk keeps appearing in first place: a !!! hint in advanced/clients.md. Here is its full text:

!!! hint
    If you are coming from Requests, `httpx.Client()` is what you can use instead of `requests.Session()`.

It's only 115 characters, contains httpx, requests and Client all at once, and is about "how to use httpx" in general. To an embedding model it looks a bit like almost any "how do I … in httpx" question. And being short, it has no other content to dilute that vague relevance, so it ranks first for many questions. With e5, the role goes to the opening of logging.md instead, also a passage that talks about httpx in general terms.

This is common: very short, general chunks tend to dominate vector search results. You can merge overly short chunks away at chunking time, or compensate with the methods in the next lesson.

Right idea, but not precise. For "what if the server requires Digest authentication", the top result is the chunk about SSL certificate verification. To the embedding model, "authentication" and "certificate verification" both belong to "security and identity checks", so their meanings are close. But the user asked about one specific authentication scheme, Digest, and that word itself is the key. We saw the same problem in Module 01, Lesson 5, when the embeddings couldn't tell httpx from requests. Vectors are good at the gist, not at pinning down a specific name.

The evaluation rule has limits too. For "which class should I use for asynchronous requests", the first result actually is async.md, but that chunk is the opening of the page and happens not to contain the word AsyncClient, so by the rule it counts as not found. Whether it should count is debatable. The stricter the rule, the lower the score, but also the more trustworthy.

Lesson 6 covers evaluation more systematically; next lesson we first tackle exact matching and crossing languages.

When do you need a vector database

Our 196 vectors sit in a numpy array using less than 1 MB of memory, and each search is one matrix multiplication, done instantly.

A dedicated vector database (such as Chroma, Qdrant, Milvus, or PostgreSQL's pgvector extension) is only worth it when you have hundreds of thousands or millions of vectors, or need these features:

• Large volumes. Comparing millions of vectors one by one is too slow; databases use approximate nearest neighbour (ANN) algorithms, trading a little accuracy for much greater speed.
• Persistence and incremental updates. When documents are constantly added, changed and deleted, rebuilding the whole matrix every time isn't practical.
• Filtering by conditions. For example "only search the 2.0 docs" or "only search documents this user is allowed to see".

At the scale of thousands or tens of thousands of chunks, numpy is enough. Save the matrix to a file with np.save and load it next time, skipping the vector computation. Start with the simplest approach and switch only when you actually run into the problems above.

Exercises

1. In vector_search.py, save the built matrix to a file with np.save, and on startup load it directly if the file exists. Compare the startup times.
2. Modify the chunking function to drop or merge chunks shorter than 200 characters, and rerun the evaluation. Is the catch-all chunk problem reduced? How much does the top-5 hit rate change?
3. Add 5 questions of your own to eval_qa.jsonl, confirming in the docs where each answer is and which keyword to use.

Self-check

1. Why does a dot product give cosine similarity once the vectors are normalised?

Cosine similarity is the dot product of two vectors divided by the product of their lengths. After normalisation every vector has length 1, so the denominator is 1 and the dot product equals the cosine similarity. That way the whole search needs only one matrix multiplication.

2. When evaluating retrieval, why judge a hit by "file + keyword" rather than chunk number?

Chunk numbers depend on the chunking method; change it and they all change. File names and keywords don't depend on the chunking method, so the same evaluation set can compare different chunking and retrieval methods.

3. What is a "catch-all chunk", and why does it rank so high in vector search?

A very short chunk with very general content, such as a one-line hint saying "use httpx.Client() instead of requests.Session()". It is a little relevant to many questions and has no other content to dilute that relevance, so it ranks near the top for lots of different questions, pushing out the chunks that are actually relevant.