Nobody ships a RAG system with a labelled evaluation set. Nobody should wait for one either, because a perfect dataset is not the thing that makes the evaluation worth running.
The minimum
Forty to sixty questions, written by hand, and for each one two things: the question as a user would type it, and the document that answers it. That is all. It takes an afternoon, it does not need to be complete, and it will be wrong in places, which is fine because the same set is also the test set for the next change.
The temptation is to wait for fifty clean pairs, and the reason that never happens is that the moment you write the questions, you find three of them are ambiguous. Write them anyway.
The three numbers
Recall at k. Of the questions, how often does the correct document appear in the first k results? At k = 5 this is the number that tells you whether your retrieval is the problem. If the answer is under half, everything downstream is decoration.
Faithfulness. For each answer the system produced, is every claim in it supported by the retrieved text? This is the one that catches a system that is confidently wrong, and it is the one users notice.
Latency. Time to the first token and the total time. A pipeline that is accurate and takes nine seconds is a pipeline people stop using, and the number belongs next to the quality numbers rather than in a different report.
On using a model as the judge
A model reading the retrieved text and the answer and saying whether the answer is supported is good enough to catch a regression, and it is not good enough to be the final word. It has the same failure mode as the system under test: it says yes when the text is plausible.
Two ways to keep it honest. The first is to read a sample yourself: twenty answers, by hand, once a month. The second is to change the question, not the answer: a faithful answer about the wrong document is a retrieval bug, and a judge that only sees the answer will call it faithful.
/tools/ai-rag-evaluator runs the three numbers over a set of questions and documents and gives you the table to compare two runs. /learn/llm-evaluation is the background.
What to do with the result
The output is a list of questions and the position of the right document in the returned results. Sort it by position. The questions where the right document is at position 12 are your work; the ones at position 1 need nothing, and the ones where it is absent need a document, not a retriever.