AgentHarness exists so someone can check the numbers in the paper

A vendor's harness for reproducing a vendor's numbers

AgentHarness is the evaluation harness a model company published so that others can reproduce the benchmark results it reported for its own deep-research model, in a standard reasoning-and-acting setup.

That framing is the most important thing to understand, and the README states it directly rather than presenting the project as a neutral evaluation framework. This exists to make one set of published claims checkable. It is not a tool for comparing models fairly against each other, and reading it as one would be a mistake.

Understood correctly, that is a genuinely useful thing to publish. Benchmark numbers in model announcements are usually unreproducible, because the harness, the prompts, the judge and the tool configuration are all unstated and all move the result. Shipping the harness converts an assertion into something a sceptic can run.

The results table covers four deep-research benchmarks across four model sizes, and the figures fall away sharply with size, from the low seventies down to the low teens on one benchmark. Publishing the small variants' weak numbers alongside the strong ones is a point in the project's favour.

The audience is researchers verifying those claims, and teams evaluating whether deep-research agents are usable for their own work.

Running it means standing up a lot of infrastructure

Installation itself is one command through a modern Python workflow tool, with the interpreter pinned.

uv sync --python 3.12

The real cost is everything that command does not do. The documented setup serves the model locally through a high-throughput inference server, and the published invocation requests eight-way tensor parallelism for a thirty-five billion parameter mixture model with a context window in the hundreds of thousands of tokens. Eight-way parallelism means eight accelerators. That is a cluster, not a workstation, and it is the honest entry price for reproducing these numbers on the largest variant.

Configuration is copied from an example file and filled in.

cp .env.example .env

What that file asks for is the second cost. Beyond the agent's own model endpoint, which is any interface-compatible service, it wants credentials for a separate judge model, a search provider and a web fetching service. So a full run bills three external services in addition to whatever the model serving costs.

The judge being configured separately, with the README noting the model is pinned per benchmark in a dedicated directory, is the right decision. A judge that drifts silently invalidates comparisons across time, and pinning it per benchmark is what makes a rerun mean something.

The password on the dataset is doing real work

One step in the documented setup looks strange and is deliberate. The benchmark archive is downloaded from a public hub and extracted with a password supplied in the README.

The purpose is contamination control. Benchmark questions and answers published as plain files on the open web end up in the next model's training data, at which point the benchmark measures memorisation rather than research ability. Compressing the data behind a password keeps the contents out of casual scraping while leaving them available to anyone who reads the instructions.

It is a weak protection in a cryptographic sense and an effective one in practice, because the mechanism it defeats is automated bulk collection rather than a determined person. This has become standard practice for evaluation sets that need to stay meaningful, and seeing it here suggests the team cares whether their numbers keep measuring what they claim.

A single-item run is documented, which is the cheapest way to exercise the whole path before committing to a full evaluation.

uv run python -m benchmarks.runner.run_subprocess \
  --benchmark browsecomp \
  --pipeline react_base \
  --profile default \
  --limit 1 \
  --concurrency 1 \
  --out ./tmp/smoke

Naming the benchmark, the pipeline, the profile, a one-item limit and an output path as separate arguments describes the shape of the system: benchmarks and agent pipelines are pluggable, and the repository carries directories for plugins and workflows consistent with that.

What the numbers can and cannot establish

The README carries a results table of its own, listing four deep-research benchmarks across four published model sizes.

The figures are the vendor's own, reported for their own model family under their own configuration, and they stand alone. No competing models appear in the table, so nothing in it positions this model against anything else.

That is the limitation to carry into any use of the harness. It reproduces one vendor's setup rather than offering a level field, and a different model run through these pipelines would sit inside scaffolding built around someone else's, with a judge that vendor pinned and tool integrations that vendor configured. The pluggable structure makes such a comparison possible, and anyone attempting one would need to establish fairness rather than assume it.

Other limits are ordinary. There are no tagged releases despite a stated version of 1.0.0, so a commit should be pinned. The interpreter floor is recent. And the dependency list names two commercial model clients directly, which tells you the judge path expects one of the major providers rather than anything you host.

Where this does not fit

The hardware requirement is the first exclusion and it is severe. Anyone without a multi-accelerator machine cannot reproduce the headline results locally, and while the configuration points at any interface-compatible endpoint, running the largest variant through a hosted provider replaces a hardware bill with a token bill on a benchmark suite designed to make many tool calls.

Cost is the second. Deep-research benchmarks run agents that search, fetch and reason over long contexts, and every one of those steps bills somewhere: the model, the judge, the search service, the fetch service. The repository does not estimate what a full run costs, and that absence matters more here than for most projects, because the number is not small and is the main thing preventing someone from simply trying it.

Third, this is scoped to one model family's reproduction. There is no claim of generality and no reason to expect the pipelines to suit an agent built differently.

Finally, the dependency on external search and fetch services makes results sensitive to things nobody controls. The web changes, search rankings change, and a deep-research benchmark run today and rerun in six months is not measuring quite the same task, which is a property of the benchmark category rather than of this harness.

A neutral evaluation framework is the alternative

The alternative is a general agent evaluation framework maintained independently of any model vendor, or running the underlying benchmarks directly with your own scaffolding.

The difference is whose configuration you inherit. A neutral framework aims to treat models comparably, which is what you need to decide between them, and it will not reproduce any particular vendor's published figure because it was not built to. Running benchmarks yourself gives complete control and means the scaffolding, the judge and the tool access are all yours to get right, which is substantial work and the usual reason published numbers are not reproduced.

This harness inverts that. It reproduces one setup exactly, which is worth more than a neutral framework when the question is whether a specific claim holds, and worth less when the question is which model to use.

Use this to check the claims it was published to support, which is a legitimate and underserved purpose. Use something neutral when choosing between models. And treat the two as answering different questions rather than as competing tools, because a harness that reproduces a vendor's numbers is doing its job precisely by not being neutral.

Apache terms and the first thing to settle

The project is Apache-2.0 licensed with the file present, carrying an express patent grant and permitting commercial use, which is the permissive end for a vendor-published tool. The model weights and the benchmark datasets are distributed separately and carry their own terms, which for an evaluation harness is where the meaningful restrictions usually sit. This is not legal advice.

The repository is arranged as a working system rather than a demonstration: the harness package, a benchmarks directory including the pinned judges, plugins, workflows, scripts, a lock file and an example environment.

Before starting, settle the budget question, because it is the one that stops most people. Count what a full run bills across model serving, the judge, the search provider and the web fetch service, and decide whether the single-item run documented above is as far as you need to go. Running one item costs very little and answers a different question from reproducing the table, and for many readers the first question is the useful one.

Editorial conclusion

AgentHarness is worth running if you want to check whether a specific set of published deep-research benchmark figures holds up, which is a purpose almost no model release supports and this one does. It is the wrong tool for choosing between models, since it reproduces one vendor's configuration with judges and tool integrations that vendor selected rather than offering a level comparison. Count the bill before committing, because a full run pays for model serving, a separate judge model, a search provider and a web fetch service, and note that the documented serving command requests eight-way tensor parallelism, so the headline results assume a multi-accelerator machine rather than a workstation.

Frequently asked questions

Official sources

1. ApodexAI/AgentHarness on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README