# openbench: provider-agnostic evaluation infrastructure for language models

> openbench is a Python package that runs standardized language model evaluations across dozens of suites, on any provider, with the same harness Inspect AI provides. It is installed with uv, driven by a bench command, and it can also run evaluations you keep local.

**groq/openbench** — Provider-agnostic, open-source evaluation infrastructure for language models

- Repository: https://github.com/groq/openbench
- Website: https://openbench.dev
- Stars: 827 · Forks: 107
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/groq-openbench

## What openbench is for

openbench describes itself as provider-agnostic, open-source evaluation infrastructure for language models, and the homepage is openbench.dev.

The problem it addresses is easy to state. A team that evaluates models across OpenAI, Anthropic, Google, a local vLLM deployment and a couple of smaller providers ends up with a different harness for each, and the scores are not comparable because the prompts, the parsing and the scoring differ. openbench provides standardized, reproducible benchmarking across more than thirty evaluation suites, spanning knowledge, math, reasoning, coding, science, reading comprehension, health, long-context recall and graph reasoning, and it runs the same way regardless of which provider serves the model.

The privacy angle is the other half of the pitch, and it is the part that distinguishes it from most hosted eval tools. The README calls out first-class support for local evaluations to preserve privacy, and the command list includes a way to run an evaluation from a path. A team with proprietary test data, or with data that cannot leave the organisation, can run the same harness over it.

The project is published by Groq, with Aarush Sah named as an author alongside the Groq organisation, and the repository topic tag is managed-by-terraform, which tells you something about how the team's own evaluation infrastructure is run.

## The sixty second path to a first evaluation

The quickstart is deliberately short, and the only prerequisite is uv.

```bash
# Create a virtual environment and install openbench (30 seconds)
uv venv
source .venv/bin/activate
uv pip install openbench

# Set your API key (any provider!)
export GROQ_API_KEY=your_key  # or OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.

# Run your first eval (3 seconds)
bench eval mmlu --model groq/openai/gpt-oss-120b --limit 10

# That's it! 🎉 Check results in ./logs/ or view them in an interactive UI:
bench view
```

The shape of that command is the API of the whole tool. The benchmark name comes first, the model is given as a provider and model name pair, and --limit caps the number of samples so the first run is cheap. The comment about the API key makes the provider-agnostic point concretely: the same command runs against Groq, OpenAI or Anthropic with a different key exported.

The results land in a logs/ directory, and `bench view` opens an interactive interface over them. The dual output is worth noting, because a log directory is what a CI job can archive and diff, while the interactive view is what a person reads when a score looks wrong.

The command is also available under the name openbench, since the pyproject declares both as entry points pointing at the same main function. The package name on PyPI is openbench, and the project requires Python 3.10 or newer.

## The bench command surface

The commands are the tool's real interface, and the README lists them with their purposes.

`bench list` lists the available benchmarks, and `bench describe` explains one. `bench eval <benchmark>` runs an evaluation, and `bench eval-retry <log_files>` retries a failed one, which matters for the many evaluations that fail because of a rate limit or a transient provider error rather than because of a model. `bench view` is the interactive UI over the logs, and `bench cache` manages OpenBench caches with the subcommands info, ls, clear and upload.

The README also names the model and task argument flags, -M and -T, and a --debug mode for the eval retry, plus an --alpha flag for experimental benchmarks. The --alpha flag is a small thing with a real meaning: it separates benchmarks the project considers stable from ones still being worked on, which is how you avoid a number changing under you because a task definition was revised.

The full option list is in the docs at openbench.dev/cli/overview and in `bench --help`. The README's table of common eval configuration options is cut off in the version available here, so the details of those flags have to be read in the documentation rather than in this page.

## Provider coverage and how a model is named

The provider table is the most useful part of the README for planning an evaluation, because it gives both the environment variable and the model string prefix for each one.

Each provider follows the same pattern. The environment variable is named after the provider, with the exception of Hugging Face, which uses `HF_TOKEN`, and Ollama and vLLM, which need no key because they are local. The model string is the provider name, a slash, and the model name, so `anthropic/model-name` or `openai/model-name`.

The list is long. It includes AI21 Labs, Anthropic, AWS Bedrock with AWS credentials, Azure under `AZURE_OPENAI_API_KEY`, Baseten, Cerebras, Cohere, Crusoe, DeepInfra, Friendli under `FRIENDLI_TOKEN`, Google, Groq, Helicone, Hugging Face, Hyperbolic, Lambda, MiniMax, Mistral, Moonshot, Nebius, Nous Research, Novita AI, Ollama, OpenAI, OpenRouter, Parasail, Perplexity, Reka, SambaNova, SiliconFlow, Together AI, the Vercel AI Gateway, Weights and Biases Inference and vLLM.

Two entries in that list are worth a second look. Helicone is a gateway rather than a model host, and the Vercel AI Gateway is the same, so those rows are about routing through someone else's endpoint rather than about a new set of weights. The consequence for evaluation is that the same model can be measured through different intermediaries, and openbench's naming scheme does not distinguish a provider's own endpoint from a gateway's, so a reproducibility note has to record which route was used.

## Where openbench is the wrong tool

The first limitation is the dependency pinning. The pyproject pins inspect-ai at exactly 0.3.141, inspect_swe at 0.2.26 or later, and numpy at 2.2.6 exactly. An exact pin on the framework the whole tool is built on means an upgrade is a deliberate act, and if the framework needs a security fix the upgrade arrives with whatever else changed.

The second is the size of the provider surface. The dependencies include the Groq, Anthropic and OpenAI client libraries, plus datasets, pillow, jsonschema, mcp, pydantic-settings, scipy, tiktoken, typer and platformdirs. A team that only ever evaluates against one provider still installs clients for all of them, and the local provider paths bring their own weight.

The third is the benchmark count versus benchmark depth. The feature list claims 95 or more benchmarks, and the introduction says more than thirty evaluation suites, and both figures can be true because a suite such as MMLU covers many subtasks. The number is a breadth measure, not a precision one, and a benchmark you have not read is a number without context.

The fourth is the churn risk in the benchmark set itself. The project is at version 0.5.3, with 0.5.2 and 0.5.1 both released on 2025-10-16, which is the pattern of an actively iterated project. That is good for a tool and bad for a score published in a paper, unless you pin the version and keep the logs, which is what the log directory and the eval-retry command make possible.

The fifth is the maturity of the release history. The recent releases are v0.5.3 from 2025-12-09, v0.5.2 from 2025-10-16 and v0.5.1 from 2025-10-16, so the last tag is from a while before the last push, and the project is under version 1.0.

## openbench compared with Inspect AI, HELM and a provider's own eval script

Inspect AI is the parent rather than a competitor, and the relationship is stated in the README: openbench is built on Inspect AI, described as an industry-standard evaluation framework. The pyproject makes the integration formal with an inspect_ai entry point, and the registry module registers openbench's tasks with the framework. What openbench adds is a curated set of benchmark definitions, a consistent CLI, a cache, the log viewer and the provider configuration, so a user does not have to assemble those.

HELM is the other comparison. HELM is a framework for holistic evaluation of language models, and it differs in approach because it is built around scenarios and a broad set of metrics with heavy emphasis on reproducibility and on documenting the conditions of a run, including the prompts and the exact model version. openbench is built around suites of tasks and scores, with reproducibility handled by pinning versions and keeping logs. For a leaderboard-style comparison, openbench's model of many tasks against a named model is the more direct fit; for a study that needs to characterise behaviour rather than rank models, HELM's scenario framing is the more appropriate one.

The third comparison is a provider's own evaluation script, which is what most teams start with. It is one file, it runs in seconds, and it needs no dependency. The difference is that it is not comparable with anyone else's numbers, does not handle a provider switch without edits, and usually has no log format you can diff between runs. openbench's case is that the standardisation is worth the setup once you are comparing more than one thing.

## Licence, local evaluations and the operational surface

The licence is MIT, with the terms in the LICENSE.md file, and the pyproject declares the same with a license field. The package is on PyPI as openbench, and the README badge names Python 3.10 or newer as the requirement.

The local evaluation support is the feature that changes what the tool can be used for. A privatised benchmark, one that cannot be uploaded to a public dataset service, can be run with `bench eval <path>`, and the same harness, the same scoring and the same logs apply. For a company with an internal eval set, that is the difference between having comparable internal numbers and having none.

The results can be pushed to Hugging Face datasets, which is the counterpart to the local option: a private set stays local, a public one goes to a dataset, and both produce the same log format.

The rest of the operational surface is modest. The pyproject exposes only two console scripts, bench and openbench, both pointing at the same entry point, so there is no separate server or agent to run. There is a pre-commit config, a release-please config and manifest, a CHANGELOG.md, a CITATION.cff, a CODEOWNERS file and a packages/ directory, and the source lives under src/ with the package discovery limited to openbench* under that root. The pytest configuration marks tests that need external services as integration, so they can be deselected with the marker filter, which is a sensible arrangement for a test suite that calls paid APIs.

The one operational caution is the API spend. Evaluations call paid endpoints, and a run without a limit can be expensive, which is why --limit exists and why it is in the quickstart.

## Conclusion

openbench suits a team that wants one evaluation harness across several model providers, so that a score means the same thing whichever vendor served the tokens, and it suits teams with private evaluation data that cannot be uploaded. It is the wrong tool for a one-off question about one model on one benchmark, where a single provider's own tooling is faster, and the wrong choice if you need the newest benchmark released yesterday. Before adopting it, read the CLI overview for the eval configuration options, since the README's option table stops partway through, and pin the inspect-ai version, which the dependency list fixes exactly at 0.3.141.

## FAQ

### How do I install openbench and run my first evaluation?

Install uv, create a virtual environment with uv venv, activate it, then run uv pip install openbench. Export the API key for whichever provider you want and run bench eval mmlu --model groq/openai/gpt-oss-120b --limit 10. Results land in ./logs/ and bench view opens them in an interactive interface.

### Which model providers does openbench support?

More than thirty through Inspect AI, including Anthropic, OpenAI, Google, Groq, AWS Bedrock, Azure, Cohere, Mistral, Together AI, Hugging Face, local Ollama and vLLM deployments, and gateways such as Helicone and the Vercel AI Gateway. Each provider has its own environment variable and a provider-prefixed model string.

### Can openbench run my own private evaluation data?

Yes. The README describes first-class support for local evaluations to preserve privacy, and the command is bench eval <path>, which runs an evaluation from a local path using the same harness and scoring.

### What is the difference between openbench and Inspect AI?

openbench is built on Inspect AI, which it describes as an industry-standard evaluation framework. openbench adds the curated benchmark definitions, the bench CLI, caching, the log viewer and the provider configuration, and it registers its tasks with Inspect AI through an entry point.

### How do I retry an evaluation that failed part way through?

The bench eval-retry <log_files> command retries a failed evaluation, and a --debug mode is available for it. The README also lists bench cache with info, ls, clear and upload subcommands for managing the OpenBench cache.

### What licence is openbench released under?

MIT, with the terms in the LICENSE.md file at the root of the repository, and the same licence declared in the pyproject.toml metadata.

## Sources

- [groq/openbench on GitHub](https://github.com/groq/openbench)
- [License: MIT](https://github.com/groq/openbench/blob/main/LICENSE)
- [Project website](https://openbench.dev)
- [README](https://github.com/groq/openbench/blob/main/README.md)
- [Releases](https://github.com/groq/openbench/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/groq-openbench