DeepResearch Bench: Evaluating Deep Research Agents with 100 PhD-Level Tasks

The gap DeepResearch Bench was built to fill

Deep research agents produce long, citation-heavy reports, and there is no obvious way to score one. Exact-match QA benchmarks do not apply, because the output is prose with references rather than a short answer. Human review does not scale past a handful of runs. The project's stated motivation is exactly this: the README says the benchmark addresses "the absence of a comprehensive benchmark for systematically evaluating Deep Research Agents (DRAs)." The intended audience is narrow. It is for teams that have built a DRA or are choosing between DRAs and want a comparable number, and for researchers who need a fixed task set to report against. It is not a tool for evaluating a single report you already have, and it is not a writing-quality rubric you can lift for internal use. The task set is 100 research tasks written by domain experts across 22 fields, described as PhD-level. That provenance matters more than the count. A hundred expert-authored tasks is a deliberate trade: small enough that each task can be checked, large enough that a single lucky run does not dominate the score.

RACE, FACT, and the split between writing and citation scoring

The evaluation pipeline has two named components. RACE is the evaluator that produces the headline scores: Overall, PAR, OPC, FAP and FAS. FACT is a separate pipeline that handles factual and citation checking, and the May 2026 notes pair it with a smaller model (GPT-5.4-mini) while RACE runs on GPT-5.5. That split is the most interesting design decision in the project. Rather than asking one judge to rate a whole report, the pipeline separates the question of whether the writing and analysis hold up from the question of whether the claims are supported. The RACE sub-scores are not explained in the README excerpt, so what PAR, OPC, FAP and FAS each measure cannot be confirmed from the supplied material; the paper at arxiv.org/abs/2506.11763 is where that would be defined. The 11 May 2026 release also changed the input handling: the notes describe "refined article-cleaning logic, using a chunk-based strategy to better support very long articles." That is a real constraint showing through. Long-form agent output has to be cleaned and chunked before a judge model can consume it, and the previous approach evidently did not handle very long articles well.

Why the evaluator swap is the story, not a footnote

On 11 May 2026 the project switched its official evaluator from Gemini-2.5-Pro to GPT-5.5, following Google's announced 17 June 2026 deprecation of Gemini-2.5-Pro. The justification is unusually concrete. The maintainers benchmarked three candidate evaluators on a human-annotated subset of 50 tasks times 4 target DRAs, giving 200 articles, and measured each candidate against human judgments. Human inter-annotator agreement was 68.78 percent. All three candidates exceeded that baseline by 1.3 to 3 points, and GPT-5.5 won on Overall, PAR and FAS. Read that number carefully. The judge agrees with humans slightly more often than two humans agree with each other, which is a reasonable bar for this kind of evaluation and also a reminder that the ceiling is not near 100 percent. The migration plan has teeth. Through 31 May 2026 there is a dual-acceptance window with two separate leaderboards so rankings stay comparable within each evaluator. By 1 June 2026 the original paper's results are re-evaluated under GPT-5.5 and migrated automatically, while prior community submissions under Gemini-2.5-Pro must be re-submitted to stay on the new leaderboard. After that date, Gemini-2.5-Pro acceptance ends. The practical consequence: a score from the old leaderboard and a score from the new one are not the same measurement, and the project is explicit about that rather than papering over it.

Getting the pipeline running

The README points to an API Configuration section for the keys and config that new submitters should follow, and the repository is Python with Apache-2.0 declared in the metadata (the README badge and the LICENSE file are the two places the licence shows up, and they do not agree, which is worth resolving before you depend on it). The legacy Gemini-2.5-Pro and Gemini-2.5-Flash evaluation code is preserved on a branch named Gemini-2.5 rather than in main, so the first thing to decide is which branch you are working from. If you want the current evaluator, you are on main and you need credentials for the GPT-5.5 judge plus GPT-5.4-mini for the FACT pipeline. If you want to reproduce an older published number, you need the Gemini-2.5 branch and Gemini credentials. Because no releases have been tagged, there is no version pin to fall back on; you are tracking a branch. The dataset is hosted separately on Hugging Face under muset-ai/DeepResearch-Bench-Dataset, and results are submitted to a leaderboard space rather than produced locally as the end state. The README does not give a full command sequence in the excerpt provided, so exact invocation has to come from the repository itself.

Where the benchmark is the wrong instrument

The evaluator dependency is the main failure mode. Every score this benchmark produces is a function of a proprietary judge model that can be deprecated, as Gemini-2.5-Pro was, or changed underneath you. The project's own migration shows the cost: re-evaluating prior results, running a dual-leaderboard window, asking community submitters to resubmit. If your goal is a stable metric you can track across a year of internal development, an LLM-judged benchmark with a rotating judge is a poor fit unless you freeze the judge version yourself and accept that your numbers will not be comparable to the public leaderboard. There is a second, quieter limitation. The task set is fixed at 100 tasks across 22 domains. That is enough to rank agents with clearly different capability levels and not enough to make fine distinctions between two agents that differ by a point or two, especially when the judge's agreement with humans sits near 69 percent. Treat small leaderboard gaps as noise. A third case: if you need to evaluate a report in a language other than English, or a domain outside the 22 covered, the benchmark cannot tell you anything, and submitting it anyway would produce a number that looks authoritative and is not.

DeepResearch Bench II and what changed in the follow-up

The natural alternative is the project's own successor rather than an outside competitor. DeepResearch Bench II was released on 6 February 2026, with its own repository at imlrz/DeepResearch-Bench-II and a paper at arxiv.org/abs/2601.08536. The difference in approach is the scoring unit. DRB II evaluates reports against 9,430 fine-grained binary rubrics covering information recall, analysis and presentation, derived from expert-written articles. DRB uses a smaller set of judge-produced sub-scores. Binary rubrics are more work to construct and more brittle to write, but they are also more auditable: you can point at the specific rubric a report failed, which you cannot do with a single holistic judge score. The maintainers are explicit that DRB II has a different evaluation focus and that DRB "will continue to be maintained and updated" alongside it. So the choice is not successor-replaces-predecessor. If you want a single comparable number against a public leaderboard, DRB is the one with the leaderboard and the migration plan. If you want to know why a report scored badly, DRB II's rubric structure is the more informative instrument. Note also the adjacent benchmarks from the same lab listed in the news section, including Wiki Live Challenge and WildGraphBench, which target writing quality against Wikipedia Good Articles and GraphRAG over long documents respectively.

Maintenance, licence, and what to check before adopting

The repository was last pushed on 11 May 2026 and has no tagged releases, so upgrade cost is not a version-diff exercise. You are following main or a named branch. When the maintainers change the judge, as they did in May 2026, your historical scores stop being comparable to new ones, and there is no release artifact to pin that would have protected you. Budget for re-running your agent whenever the evaluator changes, and for storing the evaluator identity alongside every score you record. On licensing, the metadata says Apache-2.0 while the README badge says MIT. Those are different terms, and the discrepancy is not something to guess at. Check the LICENSE file in the repository directly and, if the answer matters to how you redistribute anything derived from this, get it resolved with the maintainers or with counsel. The dataset lives on Hugging Face and may carry its own terms separate from the code. Before committing to this benchmark, verify three things: which evaluator your target leaderboard uses, whether the API Configuration keys in the README match the judge you intend to run, and whether the 22 covered domains include the ones your agent actually operates in.

Editorial conclusion

Adopt DeepResearch Bench if you are building or comparing a deep research agent and need a task set with expert provenance rather than scraped questions. Do not adopt it as a general report-quality metric: the RACE and FACT judges are model-dependent, and the project itself is mid-migration from Gemini-2.5-Pro to GPT-5.5, with the new leaderboard still under construction as of the 11 May 2026 notes. Before you run anything, check which evaluator branch you are on, since the legacy Gemini-2.5 code lives on a separate branch and only the GPT-5.5 leaderboard is maintained going forward.

Official sources

1. Ayanami0730/deep_research_bench on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README