Two weeks of tuning the retrieval on our documentation set, and the top result was the same chunk for every single query. Every question about installation returned the paragraph about installation. Every question about a flag name returned the paragraph that happened to contain the most words.
Why it happens
Dense retrieval embeds the query and the chunks and compares them in vector space. The comparison is on meaning, and a generic chunk — the one that says "this section describes the configuration options" — is close to everything. It wins by being boring.
That is fine for a question about the topic and useless for a question about the identifier. "Which flag sets the timeout" is not a semantic question. It is a string match, and pure vector search is structurally bad at it.
The fix is to search twice and fuse
The standard answer is hybrid retrieval: run a keyword search (BM25) and a dense search, then fuse the two ranked lists. Reciprocal rank fusion is the usual way to fuse, and it is simple enough to implement in a dozen lines:
def reciprocal_rank_fusion(rankings, k=60):
scores = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
return [doc_id for doc_id, _ in sorted(scores.items(), key=lambda kv: -kv[1])]
Each list contributes by position, not by score, so the two scorers do not have to be on the same scale. A document that both lists rank highly comes out on top; a document that only the keyword search likes still gets a chance.
Every engine in our comparison set can do this: Qdrant and Milvus support sparse vectors, which is BM25 inside the same engine, and with pgvector you have Postgres full-text search next to the vectors and you combine the rankings in SQL. /compare/chroma-vs-qdrant has the pair, /learn/hybrid-search the background.
Three things that make it worse, and one that makes it better
Chunks that are too generic. If your chunker cuts on fixed token windows, a chunk can start mid-explanation and contain no subject at all. Chunking on the document's own headings keeps the section with its subject, which is also what a reader would call a paragraph. /tools/rag-text-chunker shows you the chunks before you index them, which is the cheapest way to see this.
No metadata filter. "Only the pages under /v2" is a filter, not a semantic similarity, and it removes half the corpus from the competition.
Embedding a table as prose. A table flattened into a row of cells retrieves as a bag of numbers.
What makes it better, and is the one people leave out: a reranker. Retrieve twenty candidates, then rescore them with a model that sees the query and the chunk together. A cross-encoder catches the "generic chunk wins" problem in one pass, because it can tell that a chunk which merely mentions the topic is not as good as one that answers the question. /learn/reranking explains the difference from bi-encoders, and it is usually the largest single quality gain in a RAG pipeline.
The diagnostic
Before changing anything, log the id of the top five chunks for every query. If the same id is at the top for questions that have nothing to do with each other, the ranking is broken, not the content. If the right chunk is in the list but never at the top, the problem is the reranker, or the lack of one.