XRAG: A Benchmarking Harness for Individual RAG Components
XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
At a glance
- What is it?
- XRAG is a Python evaluation framework that isolates retrievers, embeddings, LLMs and orchestrators so you can measure each piece of a RAG pipeline separately. It is useful for controlled comparisons, and awkward if you want a production serving stack.
- Who is it for?
- Adopt XRAG if you are a researcher or evaluation engineer who needs to hold a RAG pipeline fixed and vary one component at a time, and you are comfortable with a research-grade tool whose roadmap still lists Semantic Perplexity, Entropy, Self-RAG and FLARE as open items. Do not adopt it as a production retrieval service.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Component-Isolation Problem XRAG Targets
Most RAG evaluation tooling scores an end-to-end pipeline. You feed a question in, you get an answer out, and a metric tells you whether the answer was good. That number hides everything. A drop could come from the chunker, the embedding model, the retriever, the reranker, the prompt template, or the generator. XRAG exists to break that apart. Its stated purpose is to evaluate the foundational components of advanced RAG systems by dissecting each core module and measuring how different configurations and components affect overall performance. The audience is narrow and identifiable: researchers who need to attribute a performance delta to a specific module, and engineers who want to justify a retriever or embedding choice with numbers rather than intuition. The project is published as the PyPI package examinationrag, carries an Apache-2.0 licence, and the accompanying paper was accepted at ICDE 2026 according to the repository updates.
How XRAG Splits a Pipeline Into Measurable Pieces
The architecture is modular by design. Retrievers, embeddings and LLMs are pluggable components, and the README lists the retrieval methods as Vector, BM25, Hybrid and Tree-based, with keyword-based and document-summary retrieval also named in the features list. Text splitters arrived in a July 2025 update and include SemanticSplitterNodeParser, SentenceSplitterNodeParser and SentenceWindowNodeParser. Sitting above the components are orchestrators, which the README describes as the layer that organises and manages execution logic and workflow, producing agentic RAG behaviour. Five orchestrator types are named: sequential, conditional, iterative, parallel and hybrid. The update log records self-rag and adaptive-rag orchestrators added on 2025-11-05 and SIM-rag added on 2025-11-18. The evaluation side is similarly layered. Traditional metrics cover F1, EM, MRR, Hit@K, MAP and NDCG. LLM-based metrics cover Faithfulness, Relevancy and Correctness. A third tier, labelled deep evaluation, covers Contextual Precision and Recall, Hallucination and Bias, and the README attributes these to LlamaIndex, DeepEval and custom metrics. That three-tier split is the substantive design decision here: it lets you report a cheap lexical score and an expensive LLM-judged score for the same run.
Installing and Driving XRAG From the Command Line
The package is distributed on PyPI as examinationrag, so installation follows the usual pip path, and the README's installation section is the place to confirm the exact extras you need. Two entry points are documented in the material. The WebUI starts with the command xrag-cli webui, which opens an interface that walks through dataset upload and configuration, index building, RAG strategy configuration, interactive testing and comprehensive evaluation in that order. The WebUI supports benchmark datasets named as HotpotQA, DropQA and NaturalQA, plus custom dataset integration with automatic format conversion. The second entry point is a generate command, added in January 2025, which produces QA pairs from a folder of documents. Configuration is handled through a config file, and the June 2025 update notes that the configuration file was revised so parameters appear clearer, with a default config added in release v0.1.4. The README also documents API support added in January 2025, which turns XRAG into a backend service rather than only a CLI. LLM backends include OpenAI models and local models such as Qwen and LLaMA, with Ollama support added in January 2025.
Where XRAG Is the Wrong Tool
The README's own roadmap is the clearest limitation statement. Semantic Perplexity, Entropy and Semantic Entropy, Auto-J and Prometheus are all listed as unchecked items under novel evaluation metrics. Adaptive Retrieval, Multi-step Approach, Self-RAG and FLARE appear as unchecked under agent workflows, even though the update log separately records self-rag and adaptive-rag orchestrators as added. That contradiction between the roadmap checkboxes and the changelog means you should test the specific orchestrator you need rather than trusting either document. Late Chunking is also unchecked. Beyond the roadmap, the deeper constraint is that XRAG is an evaluation harness, not a serving layer. The API support turns it into a backend service, but nothing in the material describes throughput characteristics, concurrency handling, or index persistence guarantees. If your question is how to serve ten thousand queries per second, XRAG is answering a different question. The LLM-judged metrics also carry the usual cost and variance of model-based scoring, and the material does not document how the framework controls for judge-model drift across runs.
How XRAG Differs From Running Ragas or DeepEval Directly
DeepEval and Ragas are the obvious comparison points, and XRAG's relationship to them is not adversarial. The README states that XRAG's built-in evaluation models include LlamaIndex, DeepEval and custom metrics, so DeepEval is a dependency of the evaluation layer rather than a rival to it. The difference in approach is what sits above the metric. Ragas and DeepEval take question, context, answer triples and score them. XRAG takes a pipeline definition, including the splitter, the retriever, the post-processor and the orchestrator, and lets you swap those pieces while holding the dataset and the metric fixed. That is the whole value proposition: the unit of comparison is the component, not the output. If you already have a fixed pipeline and only need to know whether answers are faithful, calling DeepEval or Ragas directly is less machinery. You reach for XRAG when the pipeline itself is the variable under study.
Maintenance, Releases and Licence Terms
The release cadence visible in the material is modest and front-loaded. v0.1.2, v0.1.3 and v0.1.4 all landed within about a month between January and February 2025, and the last recorded push to the default branch is June 2026, so the repository is active rather than archived. Feature updates continued through 2025 with text splitters in July and three orchestrators between November 2025 and the ICDE 2026 acceptance. That pattern suggests a research project tracking a paper rather than a library with a compatibility guarantee. There is no stated deprecation policy in the material, and no migration notes between the 0.1.x releases, which matters because the configuration file format changed in June 2025. The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant, but it also requires that you preserve copyright and licence notices and state significant changes. This is a description of the licence text, not legal advice; if you plan to redistribute XRAG inside a product, have counsel read the NOTICE requirements.
Editorial conclusion
Adopt XRAG if you are a researcher or evaluation engineer who needs to hold a RAG pipeline fixed and vary one component at a time, and you are comfortable with a research-grade tool whose roadmap still lists Semantic Perplexity, Entropy, Self-RAG and FLARE as open items. Do not adopt it as a production retrieval service. Before committing, verify three things: that the exrag CLI subcommands you need exist in the installed version, that your target LLM is reachable through the OpenAI-compatible or Ollama path, and that the DeepEval-based metrics you intend to rely on are the ones actually wired into the config you copy.
Community notes