LiveBench: a monthly-refreshed LLM benchmark that scores answers without an LLM judge

The contamination problem LiveBench is built around

A static benchmark decays. Once questions and answers circulate, later models can be trained on them, and a high score stops meaning what it used to. LiveBench attacks this directly: the README states that questions are released monthly and that they are drawn from recently released datasets, arXiv papers, news articles, and IMDb movie synopses. The freshness of the source material is the mechanism, not a marketing angle. A model cannot have memorised an arXiv paper that appeared after its training cutoff. The second design decision matters just as much. Every question carries a verifiable, objective ground truth, which the README frames as the reason hard questions can be scored automatically without an LLM judge. That removes a whole class of disputes about whether the grader model was lenient or biased toward outputs that look like its own. The intended audience is anyone comparing models under conditions where contamination would distort the result: benchmark maintainers, evaluation teams inside labs, and engineers who need a defensible number rather than a leaderboard position. The repository also invites model submissions by issue or email, which tells you the maintainers expect to run evaluations on models they do not host themselves.

How the pipeline is organised: generation, judgment, display

The evaluation flow is split into stages rather than being a single opaque command. The README names gen_api_answer.py as the script that generates answers with API models, followed by judgment and then result display. The run_livebench.py script wraps the whole sequence: according to the documentation it handles generating answers, scoring them, and showing results in one invocation. Scoring is task-specific rather than uniform. The coding tasks are the clearest example: code_completion and code_generation require additional dependencies installed from livebench/code_runner/requirements_eval.txt, and the agentic coding questions require Docker to be installed and reachable, with the README noting that the check happens before those tasks run. The repository also states that it contains code from LiveCodeBench and IFEval, which is a plain admission that parts of the harness are inherited rather than written from scratch. That inheritance is worth knowing when you read the source: conventions from those projects may appear in the scoring paths without being documented in this README. The benchmark itself spans 18 tasks in 6 categories, and the README says harder tasks will be added over time, so the task inventory is explicitly not fixed.

Getting it running, and the flags that decide your throughput

Installation starts with a virtual environment, then an editable install from the repository root: python -m venv .venv, source .venv/bin/activate, cd LiveBench, pip install -e . For coding scores you then run cd livebench/code_runner and pip install -r requirements_eval.txt. A minimal evaluation looks like this: python run_livebench.py --model gpt-4o --bench-name live_bench/coding --livebench-release-option 2024-11-25. The flags that shape cost and speed are --parallel-requests, which sets how many questions are answered concurrently inside one task, and --mode parallel, which spins up separate tmux sessions per category or per --bench-name value. The README gives a concrete pairing: for high rate limits, --bench-name live_bench --mode parallel --parallel-requests 10; for lower limits, --mode parallel with --parallel-requests 2; for one or two tasks, drop --mode parallel and just raise --parallel-requests. --mode parallel needs tmux installed, and the README warns that the number of sessions scales with the number of categories or tasks. Other flags cover the practical realities of long runs: --resume continues an interrupted run, --retry-failures retries previously failed questions, --api-base points at an OpenAI-compatible server, and --api-key-name names the environment variable holding your key. Local inference is explicitly unmaintained; the README recommends serving the model behind a vLLM OpenAI-compatible endpoint and using run_livebench.py against it.

The 150GB Docker problem in agentic coding

This is the sharpest constraint in the material. The README states that evaluating the agentic coding tasks requires building task-specific Docker images, that storing all of them may take up to 150GB, and that the images are needed for both inference and evaluation. The README also says the team intends to optimise this later. Until that happens, the agentic coding category is effectively gated on disk capacity, not on GPU or API budget. A machine that can comfortably run the rest of the benchmark may simply not have room for this one category. The second limitation is release availability. The README states that the current release is 2025-04-25 but that not all questions for it are public on Hugging Face, and that to evaluate all categories you must pass --livebench-release-option 2024-11-25. So the freshest questions and the fully public questions are not the same set. If you report a score without stating which release you pinned, the number is ambiguous. A third limitation is structural: monthly question releases mean your historical scores are not directly comparable across releases unless you deliberately fix the release option. That is the price of contamination resistance, and it is not a bug to be worked around.

Where LiveBench is the wrong tool

If you need a stable regression suite for a model you are training, LiveBench is a poor fit. Monthly rotation means the target moves, and the README's own instruction to pin --livebench-release-option exists precisely because the default surface changes. Teams tracking a training run week over week want the same questions every time; LiveBench is designed to deny them that. It is also a poor fit for anyone without the operational tolerance for the setup. --mode parallel requires tmux, agentic coding requires a working Docker daemon, coding scores require a second dependency install, and local inference is unmaintained. A single-task evaluation on a small subset is the comfortable case; the full suite is not. Finally, LiveBench measures answer quality against objective ground truth. If your question is about subjective qualities such as tone, helpfulness, or refusal behaviour, this harness has no mechanism for it by design, and the README's insistence on avoiding an LLM judge means there is no path to adding one without departing from the project's premise.

How it differs from a static benchmark such as MMLU

The comparison is not about task difficulty, it is about the lifecycle of the questions. A static benchmark ships a fixed question set and stays frozen; its scores remain comparable across years, which is exactly why contamination accumulates against it. LiveBench inverts both properties. Questions are released monthly and sourced from recent material, so comparability across time is sacrificed for resistance to memorisation. The second difference is the grader. Static benchmarks in the multiple-choice style are scored by string match, which is cheap but shallow; harder open-ended benchmarks often reach for an LLM judge, which is flexible but introduces a model into the scoring loop. LiveBench takes the third position: hard questions with verifiable ground truth, so scoring stays automatic and judge-free. That combination is the actual contribution. If you want a frozen reference point, a static benchmark still does that job better. If you want a number that a model cannot have pre-memorised, and you accept that the number is tied to a release date, LiveBench is built for that.

Maintenance, releases, and the licence question

There are no retrieved releases for this repository, so versioning runs through the changelog file the README points to and through the dated release options accepted by --livebench-release-option. Upgrades are therefore content upgrades as much as code upgrades: pulling a newer question set changes what you are measuring. Budget for re-running evaluations when you move release dates rather than assuming scores carry over. The licence is the unresolved item. The repository metadata reports NOASSERTION, and the README's badge points at a Crates.io licence image, which is a Rust packaging artefact and does not describe this Python project. The README also states that the repository contains code from LiveCodeBench and IFEval, which means at least two upstream codebases are present inside it. Before using LiveBench inside a commercial evaluation pipeline or redistributing any part of it, read the actual LICENSE file in the repository and check the terms of those upstream projects. Nothing here is legal advice, and the metadata alone is not enough to determine the terms.

Editorial conclusion

Adopt LiveBench if you need a contamination-resistant, objectively scored signal on a model you are about to ship or compare, and you can absorb the Docker storage cost or skip the agentic coding tasks entirely. Do not adopt it if you want a stable, unchanging scoreboard, since questions rotate monthly and the release option you pass determines what you actually measure. Before running anything, verify three things: that tmux is installed if you plan to use --mode parallel, that docker --version works if the agentic coding tasks are in scope, and which release you intend to pin, because the README states the current release is 2025-04-25 while the most recent fully public questions are the 2024-11-25 set.

Official sources

1. Issues
2. LiveBench/LiveBench on GitHub
3. README