Questions3 replies
My RAG returns nothing, and I mean nothing: no results, no chunks, no documents. Not "the wrong documents" — zero.
The setup is a documentation site, crawled to Markdown, split into chunks, embedded, and stored. The collection has 4,812 chunks in it. The query goes through the same embedding model, so the dimensions match, and the search comes back empty.
I have checked the collection is not empty, that the query vector is not all zeros, and that the distances are not NaN. What else is there?
Accepted answer
Almost always one of four things, and you can tell which in two minutes.
1. The filter is excluding everything. This is the most common cause and the least suspected. If you have a payload filter on a metadata field — a tenant_id, a source, a doc_type — and the chunks you indexed do not have that field set, the filter matches nothing and the search is correct: there is genuinely nothing. Log the filter and count the chunks that pass it, before you log the results.
2. The vector is normalised differently. If the collection was created with cosine distance and you are searching with a vector that was not normalised, the distances are all wrong. Some engines require the insert vector and the query vector to be treated identically. Create the vectors the same way on both sides and check that the stored vectors have unit length.
3. The collection has a different distance or dimension than the query. You said the dimensions match, and that is worth trusting, but check the collection's own configuration rather than the model's: a collection created with a different distance will still accept a vector of the right size and return nonsense or nothing.
4. The query is outside the corpus. A query about something the documentation does not cover will return nothing, and with a strict distance threshold that is the correct answer. Test with a query you know is in the corpus — a sentence copied out of a chunk — and if that returns nothing, the problem is in the index; if it returns something, the problem is your query.
The order I would use. Test with a known-good query first. It splits the problem in half in one step, and everything after that is on one side of the split. Then check the filter, then the normalisation, then the collection configuration.
The one that wastes the most time is tuning the threshold. A threshold cannot create a result, and if the search returns nothing, the threshold is not why.
Add a fifth: the index is not built, or was not flushed. A few engines make the vector index a separate structure that is built asynchronously, and a search against a collection that is indexed but not built returns nothing while the collection reports 4,812 points.
If your engine has a "build index" or "flush" step, it is a good thing to check before anything else, because it is invisible from the query side.
It was the filter. I had a source filter set from an earlier version of the pipeline, and the new crawl wrote chunks without that field, so every query matched nothing.
The known-good query test found it in about a minute, because that sentence came straight from a chunk and it also returned nothing. Thanks — I would have spent the evening on the threshold.