OpenCompass: a hundred datasets, one warning your installer will not fail on

Version 0.4.0 moved every configuration file inside the package

There is one breaking change notice on the page and it is worth more than most changelog entries, because it invalidates paths rather than flags. In version 0.4.0, all AMOTIC configuration files, previously located in `./configs/datasets`, `./configs/models` and `./configs/summarizers`, were consolidated into the opencompass package, and users are advised to update their configuration references to reflect the change. The consequence for anyone following a tutorial written before 0.4.0 is that the command line will look right and the configuration will not resolve. When an evaluation fails on a path rather than on a score, that move is the first thing to check. It also tells you something about the project's direction: the configuration is a package asset now, not a set of files a user edits, which is a better long term shape for a platform with this many datasets and a worse one for anyone who wants to read and diff their own settings in a directory.

On Python 3.11 three code execution datasets quietly stop working

The most consequential detail in this repository is a warning inside the installer, and it is worth reading in full because it describes a feature loss rather than an error. The packaging code checks whether the requirements file ends with `requirements/extra.txt` on Python 3.11 or newer and, if so, warns that `pyext==0.7` is skipped because it depends on `inspect.getargspec`, which was removed in Python 3.11. The message then names the cost: code execution evaluation will be unavailable for APPS, with its apps and apps_mini splits, TACO, and LiveCodeBench Code Generation. So on a current interpreter you keep three datasets in the catalogue and lose the ability to run the code they ask the model to write. It arrives as a RuntimeWarning rather than a failed install, which means a CI log can carry the warning for months while a benchmark quietly reports nothing.

Installing the package downloads an NLTK corpus

The installer is not a plain one. setup.py defines a custom install command that overrides the standard behaviour:

class DownloadNLTK(install):

    def run(self):
        self.do_egg_install()
        import nltk

        nltk.download('punkt')

So a normal install performs the egg install step and then fetches the punkt corpus from the network. The same file also carries a requirements parser that reads a requirements file whose docstring says it strips specific version information, although the with_version flag in its signature defaults to True, which means the package's own dependency text is parsed by hand rather than taken from metadata. The consequences are modest but practical. In an air-gapped build the install step needs a mirror or an offline corpus. The runtime requirements are themselves a single indirection, with the top level requirements.txt containing only:

-r requirements/runtime.txt

The news list tracks provider SDKs, so your numbers move with them

The what's new section is the changelog, and reading it as one is instructive. On 2026.07.28 the platform added the OpenAI Responses API and a LiteLLM AI Gateway, and updated the Gemini and Anthropic integrations to their latest SDK interfaces, with one implementation file per provider in the models package. On 2026.08.25 it integrated with VLMEvalKit for native multimodal dataset loading, inference through OpenAI-compatible APIs and evaluation with VLMEvalKit's own metrics. Around those sit capability changes that affect interpretation rather than plumbing: multi-round inference in the inferencer plus the Multi-IF dataset on 2026.07.27, repeat analysis tools on 2026.05.25 for detecting repetitive content and looping outputs, concurrent inference with evaluation watching on 2026.03.20, and RawPromptTemplate on 2026.03.17. The consequence is that this platform follows provider libraries closely, so a provider SDK release can change how a model is called and therefore what a score means.

Judging your judge: the evaluator is a configuration choice

Two evaluator tools were added on 2025.02.15 and they define what a score actually is. `GenericLLMEvaluator` covers LLM-as-judge evaluations and `MATHVerifyEvaluator` covers mathematical reasoning assessment, and on 2025.04.01 `CascadeEvaluator` arrived, a flexible mechanism that lets multiple evaluators work in sequence for customized pipelines. Read together, they say that the number in a results file is the output of a configured chain, not a property of the model. If you use an LLM as a judge, then the judge model, its prompt and its position in the chain all feed into the score, and two projects reporting the same benchmark can be reporting different measurements. The repeat analysis tool on 2026.05.25 is the counterweight to that risk from the other direction: it exists to detect repetitive content and looping outputs.

RawPromptTemplate exists because the default templating rewrote prompts

One feature explains itself by its own name and its documentation note. `RawPromptTemplate`, introduced on 2026.03.17, allows original benchmark prompts and structured conversations to be passed to models without unintended formatting transformations. It supports API models and ChatML datasets, and it can append additional content on the model side. The phrase to notice is unintended formatting transformations, which implies that the default path does transform prompts. For an evaluation harness that is a correctness issue rather than a preference: a benchmark whose prompts are reshaped before they reach the model is no longer measuring the published benchmark, and the resulting score is not comparable with published numbers. The practical consequence is that when you compare two OpenCompass results, establish which template path produced them. Results from before this feature existed and results produced with raw prompts are different measurements wearing the same label.

Concurrent inference arrives with a watcher, a heartbeat and a queue of examples

Scale features arrived on 2026.03.20, and they arrived as a small system rather than a flag. OpenCompass supports concurrent inference across tasks together with evaluation watching, so completed inference tasks can be monitored and subsequent evaluations triggered in a coordinated pipeline, and the release describes parallel inferencers, task monitoring and heartbeat mechanisms as further improvements for large scale evaluation. The two implementations are named in the tree, one for the concurrent inferencer and one for the evaluation watcher. That is worth knowing because it is where failures move: a batch job that used to fail loudly at one task now has a watcher deciding what is finished and a heartbeat deciding what is alive. Alongside it, the repository carries example evaluation scripts in examples/, a dataset index file, an autotest directory, a tools directory with the repeat analysis script, and pre-commit configurations including a Chinese-language one.

Editorial conclusion

Adopt it when you want a reproducible harness with example evaluation scripts, provider integrations kept current, and a leaderboard you can point at. Do not treat its scores as a property of a model alone, because the evaluator configuration and the prompt template both change results. Before your first run, check the interpreter you chose against the pyext warning, decide whether you need LLM-as-judge evaluators and which judge model you are using, and read version 0.4.0's configuration move if you are following an older tutorial.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. open-compass/opencompass on GitHub
3. Project website
4. README
5. Releases