EnterpriseRAG-Bench: a synthetic corpus of 500,000 internal documents for testing RAG retrieval

The gap EnterpriseRAG-Bench was built to fill

Most public retrieval benchmarks draw on web pages, forums and documentation sites. EnterpriseRAG-Bench starts from the opposite premise. Its README states plainly that existing RAG and IR datasets focus on publicly accessible document sets such as Bing searches and Stack Overflow forums, and that no publicly accessible dataset had focused entirely on company internal data. The dataset it ships simulates a single company, Redwood Inference, an AI model inference provider, with slightly over 500,000 documents spread across nine source types: Slack at roughly 275,000 documents, Gmail at 120,000, Linear at 35,000, Google Drive at 25,000, Hubspot at 15,000, Fireflies at 10,000, GitHub at 8,000, Jira at 6,000 and Confluence at 5,000. The intended user is a team that has a working RAG pipeline and wants a target to score retrieval against before pointing it at real internal data, or a team fine-tuning an agent on enterprise-shaped content. It is not a tutorial dataset and not a drop-in knowledge base.

How the corpus and the questions are constructed

The generation process is described in methodology.md and summarised in five principles. It begins with human-in-the-loop scaffolding: a company overview, a set of initiatives, an employee directory and a source structure. A core set of high-fidelity documents is written with awareness of that context and of each other, which is what produces cross-document coherence. Higher-volume generation then falls back to topic-based scaffolding to keep diversity while controlling cost. Noise is added deliberately after the fact: document shuffling, both random and LLM-driven, miscellaneous files, and near-duplicates carrying updated or conflicting facts. Documents use project codenames, product-specific acronyms and organisational jargon, so a retriever that leans on general world knowledge will find less purchase than one tuned to the corpus vocabulary. The questions are split into ten categories: 175 Basic, 125 Semantic, 40 Intra-Document Reasoning, 40 Project Related, 30 Constrained, 20 Conflicting Info, 20 Completeness, 20 Miscellaneous, 10 High Level and 20 Info Not Found. Two of those categories, High Level and Info Not Found, have no ground truth documents at all. That is the design's most interesting choice, and also the one most likely to break a naive evaluation harness.

Getting the data and running an evaluation

The README points to two download paths: the latest GitHub release or the HuggingFace dataset page. Two archive shapes are offered. all_documents.zip contains everything in a single archive. The alternative is per-source slices named <source_type>_slice_<slice_number>.zip, each holding up to 5,000 documents with no nested directory structure. The question set lives in questions.jsonl at the repository root and is also attached to releases. A separate extra_questions.jsonl holds 100 metadata-dependent questions for teams doing metadata-aware RAG; the README notes these are excluded from the leaderboard because their evaluation criteria differ from the core retrieval-focused benchmark. For scoring, the README directs you to answer_evaluation/README.md, which covers answer evaluation only. A quickstart.md at the root collects the remaining usage paths. Everything here is file-based: you fetch archives, unzip, index them into whatever store you run, and evaluate answers against questions.jsonl. There is no server to start and no config key to set beyond your own pipeline's.

Where the benchmark stops measuring

Two limitations are visible in the material itself. First, the 100 metadata-dependent questions sit outside the leaderboard. If your production system relies on filters such as author, date range, channel or repository, the headline benchmark does not score that, and you would be running the extra set on your own with criteria the README does not define. Second, the 10 High Level and 20 Info Not Found questions have no ground truth documents, so a harness that computes recall over retrieved document IDs will score them as failures by construction. The README describes Info Not Found as questions where the answer is not available, which means the correct behaviour is abstention, and abstention is a different metric from retrieval accuracy. The Conflicting Info category, 20 questions where documents directly contradict each other, has a similar trap: the README says the system must give a complete and correct answer, and a retriever that returns one side of the contradiction will look correct on a single-document metric. Treating all 500 questions as one undifferentiated recall number will misrepresent your system in both directions.

What it does not replace: BEIR-style public benchmarks

The obvious alternative for retrieval evaluation is a public-corpus benchmark such as BEIR or the MS MARCO derived sets. The difference is not scale, it is the distribution of the documents. Public benchmarks draw on web text and forum posts, where entities are named consistently, terminology is shared with the model's pretraining data, and near-duplicates are rare. EnterpriseRAG-Bench inverts each of those properties: project codenames and acronyms that are meaningless outside the fictional company, near-duplicates with updated or conflicting facts, and misfiled documents. A retriever that scores well on a public benchmark by matching surface terms can underperform here without any code change, because the signal has moved from lexical overlap to internal context. The two are complementary rather than substitutes. Public benchmarks tell you whether your embedding model is competitive in general; EnterpriseRAG-Bench tells you whether it survives a corpus that looks like the one your users actually search.

Maintenance, licensing and what the repository commits to

The repository is MIT licensed and not archived. The dataset carries its own version, Dataset v1.0.0, released on 2026-03-29, separate from any code versioning, so a pinned download is reproducible by release tag rather than by commit. The last push to the repository was 2026-09-03. The README also states that the accompanying code provides a way to generate similar datasets for different industries and company scales, which matters for maintenance cost: if the fixed Redwood Inference corpus drifts away from your domain, regeneration is the documented escape hatch, and methodology.md is the file that governs how faithful a regenerated corpus will be. The MIT licence covers the code; the README attaches the MIT badge to the code specifically, and the dataset is distributed through GitHub releases and HuggingFace, so check the dataset card for any terms that differ before redistributing the corpus. That is a reading task, not a legal opinion, and it is worth doing before the data lands in a shared index.

The generation pipeline as the real deliverable

The 500 questions are a fixed target, and fixed targets go stale. The more durable part of this repository is the generation path described in methodology.md: scaffold a company, write a coherent core, scale with topic-based scaffolding, then inject noise. That ordering is the argument the project is making. Coherence before volume, noise after both. A team that only downloads all_documents.zip gets a static corpus. A team that reads methodology.md gets a recipe for producing a corpus whose source mix, topic spread and noise profile match its own company, which is the only way to keep the benchmark honest as the system under test changes. The README's five principles are worth reading as design constraints rather than marketing: realistic volume distribution and realistic noise are the two that most public benchmarks skip, and they are the two that most often explain why a pipeline that looks good in a demo fails in production.

Editorial conclusion

Adopt EnterpriseRAG-Bench if your retrieval stack already ingests Slack, Gmail, Linear, Drive, Hubspot, Fireflies, GitHub, Jira and Confluence style content and you need a scored target that is not the public web. Do not adopt it if you need a benchmark for metadata-filtered retrieval, multilingual corpora, or per-document access control, because the core 500 questions do not cover those and the metadata set is excluded from the leaderboard. Before committing, verify three things in the repository: that the 500 questions in questions.jsonl match the document slices you actually download, that the answer_evaluation README's scoring path fits your pipeline's output format, and that the 20 Info Not Found plus 10 High Level questions, which carry no ground truth documents, are handled by your harness as abstention cases rather than retrieval misses.

Official sources

1. License: MIT
2. onyx-dot-app/EnterpriseRAG-Bench on GitHub
3. Project website
4. README
5. Releases