open-rag-eval: TREC-RAG Metrics Without Golden Answers

The labelling problem open-rag-eval is built around

Standard RAG evaluation asks you to write a question, then write the correct answer, then decide which retrieved chunks were the right ones. That reference set is the expensive part. It has to be produced by someone who knows the domain, it goes stale as the corpus changes, and it does not scale past a few hundred queries without a labelling budget. open-rag-eval attacks that cost directly. Its README states that the core metrics, UMBRELA and AutoNuggetizer, do not require golden chunks or golden answers, and that this is what makes RAG evaluation easy and scalable. Both techniques are attributed to Jimmy Lin's lab at UWaterloo, with links to arXiv papers 2406.06519 and 2411.09607. The intended user is an engineer who already has a working retrieval and generation pipeline and wants a score they can compare across query configurations. The README also notes that golden answer evaluation is supported as an optional extra when reference answers do exist, so the toolkit is not locked into the reference-free path. If your team already maintains a labelled set, this is not the only mode you can run.

What the pipeline actually does between your CSV and the results folder

The data flow is a config file, a query list, a connector, an evaluator, and an output directory. You supply a CSV with exactly one column named query. The config names that file under input_queries, names a generated_answers path and an eval_results_file path, and sets a results_folder where all artifacts land. The connector section under options/query_config is where the retrieval and generation side is configured; for Vectara that means a corpus_key plus whatever query parameters you want to test. The evaluator then consumes the answers the connector produced. The README describes the reporting as per-query scores plus intermediate outputs for debugging and analysis, and lists plotting utilities for comparing results across configurations or runs. That intermediate output matters more than the headline number: when a run scores badly you need to see which query failed and what the judge saw. The architecture is deliberately modular, with custom metrics and connectors as the extension points, and the README names LlamaIndex and LangChain connectors as shipping today with more coming. The evaluation metrics themselves are documented separately in METRICS.md rather than in the README.

Install and the exact keys you need before a first run

Installation is a single pip command: pip install open-rag-eval. That installs the open-rag-eval command-line tool, which the README says is enough to run evaluations and plot results without cloning the repository. If you want the example configs, clone the repo, cd into it, and run pip install -e . instead. Python 3.9 or higher is required. Three credentials gate the default path. First, an OpenAI API key exported as OPENAI_API_KEY, because the default LLM judge model used in some metrics runs through OpenAI. Second, a Hugging Face token exported as HF_TOKEN, needed for the open-source HHEM hallucination model that is the default for factual consistency evaluation; you also have to request access to the vectara/hallucination_evaluation_model model page on Hugging Face before the token will work. The alternative is Vectara's commercial Factual Consistency API, which swaps the Hugging Face dependency for a Vectara API key. Third, for the Vectara connector specifically, a Vectara account, a corpus with indexed data, an API key with querying permissions, and your Customer ID and Corpus key. The config example eval_config_vectara.yaml wires these together, and the connector section reads the API key via ${oc.env:VECTARA_API_KEY}.

Prompt templates as a config value, not a code change

One detail worth calling out because it affects how you run comparisons: the Vectara connector accepts a custom prompt_template under query_config/generation. You can point it at a text file path, in which case the connector reads the file content as a string and sends it to Vectara's API, or you can inline the template as a YAML string. The README notes that files are read with UTF-8 encoding and that leading and trailing whitespace is stripped from file content. That is a small feature with a real consequence. If your evaluation is meant to isolate the effect of a prompt change, you can express both variants in config rather than in application code, which keeps the run reproducible. The config file also lets you customize any Vectara query parameter, so the unit of comparison can be a retrieval setting, a generation setting, or a prompt, all from the same YAML shape. The README does not describe how prompt templates interact with the metric implementations, so treat a prompt-only comparison as something you should sanity-check against the intermediate outputs rather than trust blindly.

Where this toolkit will not help you

The dependency chain is the first limitation. The README's own prerequisites list an OpenAI API key for the default judge and a gated Hugging Face model for hallucination detection. An air-gapped or privacy-constrained environment cannot run the default configuration without substituting the Vectara commercial API for factual consistency, and even then the judge model requirement stands unless you supply a custom metric. That is a real constraint, not a footnote. The second limitation is scope. The out-of-the-box connectors are Vectara, LlamaIndex and LangChain. If your RAG stack is something else, you are writing a connector before you evaluate anything, and the README gives no estimate of how much work that is. The third is that the metric documentation lives in METRICS.md, which is referenced but not reproduced in the README. Anyone deciding whether UMBRELA and AutoNuggetizer produce numbers they can defend to a stakeholder has to go read that file and the two arXiv papers. The README also does not state what judge model versions are pinned, which matters if you intend to compare scores across runs separated by months. Verify that yourself before treating a score as a baseline.

How it differs from a labelled-set framework like Ragas

The obvious comparison is Ragas, which also targets RAG evaluation in Python. The difference is in what each assumes you have. Ragas-style evaluation is typically built around a reference dataset: you supply questions with ground-truth answers and, often, ground-truth contexts, and the metrics compare your pipeline's output against those references. open-rag-eval inverts the assumption. Its core metrics operate without golden chunks or golden answers, which is the whole premise stated at the top of the README, and reference-based evaluation is offered as an optional addition rather than the default. That changes the adoption cost. With a reference-based framework, the first task is building the dataset. With open-rag-eval, the first task is getting three credentials working. It also changes what the score means: a reference-free metric is a judge's opinion about answer quality against retrieved evidence, not a distance from a known-correct string. If your organisation needs an auditable number tied to a curated answer key, the reference-free path will not satisfy that, and you should be using the optional golden answer metrics instead. The README does not compare the two approaches on cost or reliability, so that judgement is yours to make.

Maintenance, licensing and what to pin

The repository is Apache-2.0, which permits commercial use and modification, and it is not archived. Recent releases are v0.2.3 in October 2025, v0.2.4 in November 2025, and v0.3.0 in December 2025, with the last push to the dev branch dated June 2026. That cadence suggests active work, though the default branch being dev rather than main is worth noting if you plan to vendor a copy. Apache-2.0 also means the usual obligations apply around retaining notices and stating changes; that is a general property of the licence, not legal advice, and your own counsel should review anything you redistribute. The practical upgrade cost is the same as the adoption cost: every release can move the judge model, the HHEM model, or the metric implementations, and any of those can shift your scores without your pipeline changing. Pin the version in your requirements file and record the version alongside each results folder. The README does not document a migration path between v0.2.x and v0.3.0, so read the release notes for v0.3.0 before upgrading a baseline you care about. If you cannot find a changelog entry covering a metric change, re-run a fixed query set on both versions and diff the per-query outputs rather than the aggregate.

Editorial conclusion

Adopt open-rag-eval if you already run a RAG pipeline behind Vectara, LlamaIndex or LangChain and want per-query scores without hand-labelling a reference set. Do not adopt it if you cannot supply an OpenAI API key, cannot get access to the gated vectara/hallucination_evaluation_model on Hugging Face, or need an evaluation stack that runs fully offline. Before writing any config, confirm three things: that your corpus key and VECTARA_API_KEY work against the connector, that your HF_TOKEN has been granted access to the HHEM model page, and that the eval_config_vectara.yaml you copy actually points input_queries at a CSV whose single column is named query.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vectara/open-rag-eval on GitHub