Hybrid search and reranking

Last lesson's vector search found the right chunk for only a little over half of the 20 questions. The problems come down to two: it isn't sensitive to exact names like "Digest" or "NO_PROXY", and matching Chinese questions to English documents is inherently hard.

This lesson adds three things in turn: keyword search, query rewriting and reranking. After each one, we run the same 20 questions through last lesson's evaluate to see how much the numbers actually move. The conclusion up front: the biggest gain comes from a place you might not expect.

Keyword search: BM25

Before vector search became popular, search engines always used keyword search: the more often and the more concentrated the question's words appear in a document, the higher its score. The most widely used algorithm is BM25. It is smarter than "count the occurrences" in two ways:

• Rare words matter more. "the" is in almost every chunk, so matching it says nothing; "DigestAuth" appears in only one or two chunks, so a match is a strong signal. This weight is called IDF (inverse document frequency).
• Term frequency has diminishing returns. A word appearing 10 times in a chunk doesn't make it 5 times as relevant as appearing twice. BM25 makes the score grow more and more slowly with frequency. It also penalises very long chunks, since long chunks naturally contain more words of every kind.

Written from scratch:

import math
import re
from collections import Counter


def tokenize(text):
    # 英文单词和代码标识符按单词切（转成小写），中文按单个汉字切
    return re.findall(r"[a-z0-9_]+|[一-鿿]", text.lower())


class BM25:
    def __init__(self, chunks, k1=1.5, b=0.75):
        self.chunks = chunks
        self.docs = [tokenize(text) for _, text in chunks]
        self.avg_len = sum(len(d) for d in self.docs) / len(self.docs)
        self.tf = [Counter(d) for d in self.docs]
        df = Counter(word for d in self.docs for word in set(d))
        n = len(self.docs)
        # 越少的块里出现的词，越能说明问题，权重越高
        self.idf = {w: math.log(1 + (n - c + 0.5) / (c + 0.5)) for w, c in df.items()}
        self.k1, self.b = k1, b

    def score(self, query_words, i):
        tf, length = self.tf[i], len(self.docs[i])
        s = 0.0
        for w in query_words:
            if w in tf:
                # 词频越高分越高，但增长越来越慢；块越长，同样的词频得分越低
                s += self.idf[w] * tf[w] * (self.k1 + 1) / (tf[w] + self.k1 * (1 - self.b + self.b * length / self.avg_len))
        return s

    def search(self, query, k=5):
        words = tokenize(query)
        scores = [(self.score(words, i), i) for i in range(len(self.docs))]
        scores.sort(reverse=True)
        return [(s, *self.chunks[i]) for s, i in scores[:k]]

k1 controls how quickly term frequency saturates, b controls how strongly long chunks are penalised; 1.5 and 0.75 are the usual defaults. This implementation scores every chunk each time, which is no problem for 196 chunks; with hundreds of thousands of chunks you'd use an inverted index to score only the chunks that contain the query words, which is what search engines like Elasticsearch do. There are ready-made Python implementations too, such as the rank_bm25 package.

Searching with the Chinese question directly: poor

BM25（原问题）      第 1 名  15%  前 3 名  30%  前 5 名  35%  MRR 0.229

Much worse even than vector search (65% top-5). The reason is simple: the questions are in Chinese, the documents in English, and the words just don't match. In "服务器要求 Digest 认证怎么办？" ("what if the server requires Digest authentication?") only the one English word "digest" can match; none of the Chinese characters appear anywhere in the English docs.

Query rewriting

If the words don't match, make them match: first have the LLM rewrite the Chinese question into English search terms, then search with those.

def rewrite(question):
    """把中文问题改写成英文检索词。httpx 的文档是英文的，这样关键词才能对上。"""
    if question not in rewrites:
        response = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": (
                "把下面这个关于 Python 库 httpx 的问题，改写成用于搜索 httpx 英文文档的检索词。"
                "输出一行英文，包含问题的英文翻译，以及文档里可能出现的参数名、类名、术语。不要解释。\n\n" + question)}],
            extra_body={"thinking": {"type": "disabled"}},
        )
        rewrites[question] = response.choices[0].message.content.strip()
        CACHE.write_text(json.dumps(rewrites, ensure_ascii=False, indent=2))
    return rewrites[question]

Besides "translate into English", the prompt also asks for "parameter names, class names and terms likely to appear in the docs". An example rewrite:

服务器要求 Digest 认证怎么办？ → httpx Digest authentication server requires digest auth how to use DigestAuth parameter class terms

The model didn't just translate; it also guessed the matching httpx class name, DigestAuth. This is exactly what models are good at: it knows roughly what httpx looks like and can turn the user's casual question into the technical vocabulary the docs use.

Rewrites are cached in rewrites.json, so each question costs money only once. In a real application every user question needs one extra model call, adding a few hundred milliseconds and less than $0.0001.

Fusion: RRF

Vector search and keyword search each have their strengths: one catches the gist, the other catches exact words. Can we use both?

The difficulty is that their scores can't simply be added: vector search gives a similarity between 0 and 1, while a BM25 score might be ten or several dozen. The common approach is to ignore the scores and look only at the ranks, called RRF (Reciprocal Rank Fusion):

def rrf(result_lists, k=5, c=60):
    """倒数排名融合：一个块在每个列表里排第 r 名，就得 1/(c+r) 分，把各列表的分数加起来。"""
    scores, items = Counter(), {}
    for results in result_lists:
        for rank, (_, file, text) in enumerate(results, 1):
            scores[(file, text)] += 1 / (c + rank)
            items[(file, text)] = (file, text)
    return [(s, *items[key]) for key, s in scores.most_common(k)]

A chunk ranked high in both lists scores high; one that appears in only one list scores lower. c=60 is the usual value from the original paper; it keeps the gap between rank 1 and rank 2 from being too large. In use, each search method contributes its top 20, and we take the top 5 after fusion.

Results

code/04-rag/hybrid.py runs all these combinations, using multilingual-e5-small, the better model from last lesson, for vector search:

向量（原问题）        第 1 名  30%  前 3 名  50%  前 5 名  65%  MRR 0.418
BM25（原问题）      第 1 名  15%  前 3 名  30%  前 5 名  35%  MRR 0.229
向量（改写后）        第 1 名  55%  前 3 名  90%  前 5 名  95%  MRR 0.718
BM25（改写后）      第 1 名  70%  前 3 名  95%  前 5 名 100%  MRR 0.804
混合 RRF（改写后）    第 1 名  60%  前 3 名  90%  前 5 名 100%  MRR 0.772

Query rewriting does most of the work. For BM25 the top-5 hit rate jumps from 35% to 100%; for vector search it rises from 65% to 95%. One cheap model call helps more than any change of search algorithm. In our "Chinese questions, English docs" setting this step is practically required. Even when questions and docs are in the same language, rewriting a casual question into the docs' vocabulary usually helps.

After rewriting, BM25 beats vector search. Technical docs are full of parameter and class names, the rewritten search terms happen to contain them, and keyword search matches them at once. That runs against the common impression that "vector search is more advanced".

Hybrid search wasn't better on this question set. After RRF fusion the top-5 hit rate is 100%, the same as BM25, but MRR actually fell from 0.804 to 0.772. Vector search's poorer rankings pushed some correct answers that BM25 had ranked first further down.

This doesn't mean hybrid search is useless. On other data, for example when users' questions are very casual and the docs contain no matching keywords, vector search contributes much more. My point is: don't use a method because it sounds more advanced; let evaluation data decide. 20 questions is too few, and this difference may just be noise; with your own data the conclusion might be completely different.

Reranking

Vector search turns the question and the documents into vectors separately and then compares the vectors. At computation time the question and the document can't "see" each other. That is fast, since all document vectors can be computed in advance, but it's coarse.

A reranker works differently: it joins the question and a chunk together and hands them to the model, which reads both at once and directly outputs a relevance score. That judges much more accurately, but every pair has to be computed separately and nothing can be precomputed, so it's much slower.

So there are usually two stages: first pick a rough 20 chunks with a fast method (vector, BM25, hybrid), then have the reranker score those 20 precisely, reorder them and take the top 5.

I used BAAI's open-source bge-reranker-base, which supports Chinese and English, is a 1.1 GB download and runs on a CPU:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base", max_length=512)


def reranked(question, k=5, use_rewrite=False):
    pool = candidates(question, 20)
    query = rewrite(question) if use_rewrite else question
    scores = reranker.predict([(query, text) for _, _, text in pool])
    order = sorted(range(len(pool)), key=lambda i: -scores[i])
    return [(float(scores[i]), pool[i][1], pool[i][2]) for i in order[:k]]

candidates is the "rewrite + hybrid RRF" from above, top 20 chunks. Results (code/04-rag/rerank.py):

混合 RRF（改写后）        第 1 名  60%  前 3 名  90%  前 5 名 100%  MRR 0.772  （20 题用时 0.2 秒）
混合 + 重排（用原中文问题）    第 1 名  45%  前 3 名  85%  前 5 名  90%  MRR 0.643  （20 题用时 20.1 秒）
    没找到：怎么让请求走 HTTP 代理？（应在 advanced/proxies.md）→ 第 1 名 advanced/proxies.md
    没找到：有些域名不想走代理，环境变量怎么设置？（应在 environment_variables.md）→ 第 1 名 environment_variables.md
混合 + 重排（用改写后的问题）   第 1 名  85%  前 3 名  95%  前 5 名 100%  MRR 0.912  （20 题用时 20.0 秒）

Reranking with the rewritten English question raises the top-1 hit rate from 60% to 85% and MRR from 0.772 to 0.912, the best of all the combinations. For RAG, whether the first result is right matters a lot: models tend to weight the material placed first most heavily, and it lets you include fewer chunks, saving money and reducing distraction.

Reranking with the original Chinese question made things worse. bge-reranker-base does support both Chinese and English, but its judgement is less accurate when a Chinese question is paired with English documents. Interestingly, for the two "not found" questions, the first result is actually in the right file, just without the keyword, which means it found roughly the right place but didn't pick the most precise chunk.

The cost is time: 20 questions took 20 seconds, about 1 second each, all spent on the CPU scoring 20 candidate chunks. A GPU would be much faster; you can also reduce the number of candidates or use a reranking API service. Whether to add reranking depends on whether your application can afford that extra second.

What each step contributed

In your own project, try things in this order: start with the simplest single search method and build an evaluation set; then try query rewriting; then hybrid; and consider reranking last. Look at the numbers at every step, and don't add anything that doesn't help.

Exercises

1. Change BM25's k1 to 0.5 and 3, and b to 0 and 1, and see how the "BM25 (rewritten)" results change.
2. Modify the rewrite prompt to ask only for translation, not for parameter and class names, and rerun it (delete rewrites.json first). How much does the top-5 hit rate drop?
3. Give the two search methods different weights in rrf, say doubling BM25's score, and see whether hybrid search can beat BM25 alone.
4. Change the number of rerank candidates from 20 to 10 and see how accuracy and time change.

Self-check

1. Why does BM25 give rare words more weight?

A word found in almost every chunk (like "the" or "httpx") can't tell you which chunk is more relevant when it matches; a word found in only a few chunks (like "DigestAuth") strongly indicates that the chunk is related to the question when it matches. IDF measures how rare a word is.

2. Why can't you just add vector search and BM25 scores together? How does RRF solve this?

Their score ranges are completely different: vector similarity is between 0 and 1, a BM25 score can be dozens, so adding them directly lets BM25 dominate. RRF ignores the raw scores and uses only each result's rank in its own list, scoring 1/(c + rank) and summing, so the two methods' rankings combine fairly.

3. A reranker is more accurate than vector search. Why not use it to search all the chunks directly?

A reranker has to process the question together with each chunk, one computation per chunk, with nothing precomputed. With many chunks that's far too slow: 196 chunks means 196 computations, and hundreds of thousands is out of the question. So a fast method first picks a few dozen candidates, and the reranker then orders them precisely.