lmms-eval and the bet on one pipeline for multimodal benchmarks

The problem lmms-eval names is two teams, one model, two numbers

Benchmark disagreement is the target, not multimodal capability. The README argues that the evaluation ecosystem is fragmented: scattered datasets, inconsistent post-processing, and single accuracy numbers that hide whether a gain is real or random, so that two teams running the same model on the same benchmark routinely publish different results. If you have ever tried to explain why your MME score is three points below a leaderboard entry, this is the toolkit built around that problem.

The audience is a model team with a reason to defend numbers: researchers comparing checkpoints, and engineers who have to say why a candidate beats the incumbent. A hobbyist with one model and one dataset will find far more machinery than the job needs.

Three principles are stated outright: reproducible (one pipeline, deterministic results, same model and benchmark producing the same numbers every time), efficient (async serving, adaptive batching, video I/O work), and trustworthy, which is where the interesting claim sits. The toolkit reports confidence intervals, clustered standard errors and paired comparisons instead of a bare accuracy figure. That is the part most evaluation libraries skip, and it is the part that decides whether a score can carry a release decision.

Everything reaches one module, lmms_eval.__main__, and every task is a YAML file

The architecture is a single Python module entry point fed by flags. Model code lives under lmms_eval/models, and the README points at more than 30 backends there. Tasks are configuration, not code: the quickstart selects one by name, and the alternative install path carries a comment that you might need to add your own task yaml when installing straight from Git rather than cloning.

Recent work has landed as named mechanisms rather than themes: agentic task evaluation through generate_until_agentic, a video I/O overhaul with TorchCodec the maintainers call up to 3.58x faster, per-sample token counts and run-level throughput as explicit efficiency metrics, and flattened JSONL log output meant to be easier to post-analyse. An earlier step added a standalone HTTP eval server, described as roughly 7.5x the throughput of the line before it, plus paired t-test support.

The dependency list in pyproject.toml reads like the architecture does. torch and torchvision, av, opencv-python-headless and yt-dlp cover multimodal input. datasets, evaluate, jsonlines, sacrebleu and sklearn cover scoring and caching. pycocoevalcap and latex2sympy2-extended are there for specific benchmark families rather than general use, and chess is pinned for MET-Bench board-state scoring. requires-python is set to >=3.10.

uv is the sanctioned path, and the README recommends it for consistency

The project standardises on uv so that everyone ends up with the same package versions. The installer is a one-liner:

curl -LsSf https://astral.sh/uv/install.sh | sh

After that, clone and install in editable mode with the all extra:

git clone https://github.com/EvolvingLMMs-Lab/lmms-eval
cd lmms-eval
uv pip install -e ".[all]"

Commands then run under the environment wrapper:

uv run python -m lmms_eval --help

An alternative install exists for people who want the package without a working tree. It creates a named virtualenv, pins the interpreter, activates it, and installs straight from the Git URL. Read the warning in that snippet before choosing it: you may have to supply your own task yaml, which means the config files shipped in the repository are not guaranteed to be on your path.

uv venv eval
uv venv --python 3.12
source eval/bin/activate
uv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git

There is also a commented uv sync line, which the README describes as creating or updating the environment from uv.lock. The repository does ship a uv.lock.

The smoke test is eight samples of Qwen2.5-VL on MME with batch size 1

The first run is deliberately small, and the flags tell you what a real run looks like. This is the README's quickstart command verbatim:

python -m lmms_eval \
  --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct \
  --tasks mme \
  --batch_size 1 \
  --limit 8

--limit 8 caps the number of samples, so it tells you nothing about quality. The checkpoint identifier travels inside --model_args as key=value rather than as its own flag, which is the pattern to learn before you touch anything else.

The success condition is deliberately unambitious: if metrics print, the environment is ready. Nothing about download caching, Hugging Face token handling or offline use is described here, so a first run will hit the network for the model weights and the task data.

One dependency is conditional rather than optional-by-choice. Caption benchmarks such as coco, refcoco and nocaps need pycocoevalcap to talk to Java, and the README states you need java==1.8.0 for that, installable with conda and checkable with java -version.

conda install openjdk=8

That is a Java 8 requirement sitting inside a Python 3.10+ project, and it is the sort of detail that only surfaces on the benchmark you actually want.

Determinism is a promise the README qualifies with its own torch caveat

The reproducibility claim is the strongest thing the project asserts: same model, same benchmark, same numbers, every time. Set against that, the LLaVA-1.5 reproduction note is the most useful sentence in the README. It points at miscs/repr_torch_envs.txt and miscs/llava_result_check.md, and it says plainly that torch and cuda version differences cause small variations in the results.

Read together, those two parts pull against each other. The pipeline is deterministic given a fixed environment; it is not portable across environments. For a team whose whole reason for adopting this harness is comparable numbers, that distinction decides whether the tool works. Pin the torch and cuda versions from miscs/repr_torch_envs.txt into your own environment record, or the reproducibility claim will not carry the weight you put on it.

The last push was on 2026-11-02, and the harness carries an MIT licence declared in pyproject.toml with a LICENSE file at the root.

The av pin and the 3.12 venv hint at a resolver you will fight at least once

Two details are worth reading together before installing on a clean machine. requires-python is >=3.10, but the dependency on av is pinned as av>=17.1.0,<18.0.0 with an inline comment reading "PyAV 18 requires Python 3.11+". The alternative install path then creates its virtualenv with uv venv --python 3.12. The supported floor and the recommended interpreter are not the same interpreter.

The upper bound av<18.0.0 applies on every interpreter, including 3.12, so the same PyAV line is installed whichever supported Python you pick.

Other entries carry similar comments: torch>=2.1.0 is annotated as the way to enable sdpa mode for running a 34B model on one 80GB GPU.

One more trap for evaluation work: the LLaVA-1.5 reproduction note records that torch and cuda version differences cause small variations in results. A number from this harness is only comparable to another number produced on a matching stack.

Where VLMEvalKit draws the line differently

The comparison that matters is with VLMEvalKit, and the difference is architectural rather than a matter of quality. lmms-eval is built around a CLI module with model adapters and YAML task definitions, and it can hand evaluation off to a standalone HTTP server rather than running in your shell. The emphasis on confidence intervals, clustered standard errors and paired comparisons puts lmms-eval on the side of statistical defensibility: it is built for the argument after the benchmark, not only the number from it.

If your need is throughput over many models in a controlled batch and you already have an evaluation stack you trust, a harness organised differently around model serving and result aggregation may fit better. lmms-eval's own cost is visible in the dependency list: installing with the all extra pulls torch, opencv, PyAV, yt-dlp, pycocoevalcap and sklearn whether or not you evaluate video or captions. The modality breadth that makes it useful is the same breadth you pay to install.

The honest summary is that lmms-eval optimises for a claim you intend to defend, and pays for it in setup weight.

Editorial conclusion

Adopt lmms-eval if you evaluate more than one modality, or if a reviewer will ask for a paired t-test rather than a single accuracy figure. Leave it if your work is one caption dataset, where the Java 8 requirement for pycocoevalcap costs more than the statistics are worth. Verify the install first: run uv pip install -e ".[all]" on Python 3.12 and confirm the av pin resolves, because pyproject.toml states requires-python >=3.10 while holding av below 18.0.0 and the alternative install path creates its virtualenv with uv venv --python 3.12. Then pin the torch and cuda versions recorded in miscs/repr_torch_envs.txt into your own environment record, since the project states those differences move LLaVA-1.5 scores.

Frequently asked questions

Official sources

1. EvolvingLMMs-Lab/lmms-eval on GitHub
2. Issues
3. Project website
4. README
5. Releases