AISBench: model evaluation built on the OpenCompass conventions

Why an OpenCompass compatible harness exists

AISBench is described in the repository description as a model evaluation tool built on OpenCompass. The key word is compatible: the tool is built on the configuration system, the dataset structure and the model backend implementation of OpenCompass, while extending support for service based models.

That framing is a decision about migration cost. Evaluation harnesses are sticky, because a team that has tuned prompts, wired in its own datasets and built dashboards around OpenCompass does not want to redo that work to test a model that is now only available as an HTTP endpoint. Reusing the same configuration format and dataset layout means an existing OpenCompass setup can be pointed at a served model without being rewritten.

The gap being filled is service based models. Most evaluation tooling assumes a model you can load in the same process as the harness. A model behind an API endpoint, with its own queueing, its own rate limits and its own cache, breaks that assumption: throughput has to be requested rather than scheduled, and the results arrive asynchronously.

The project answers that with a request scheduling layer and by distinguishing local models, configured with attr="local", from service models, configured with attr="service". The distinction shows up in the command line too. A dedicated api_model_args command group lets you pass path, model, request_rate, host_ip, host_port, url, max_out_len and generation_kwargs explicitly on the command line, and those values override the matching fields in the service model configuration during config loading. For local models the same arguments are ignored, with a warning printed so you know why nothing happened.

Installing from PyPI or pulling a container image

Since 2026-06-15 the tool has been published to PyPI, and the README gives two install commands, one plain and one with the full extra.

pip install ais_bench_benchmark

The full variant pulls the larger dependency set, which matters because the extras cover the parts of the harness that are not needed for a basic accuracy run.

The container route is documented separately in the Docker overview, and the 2026-06-15 entry is specific about the coverage: the images support x86_64 and aarch64, across Ubuntu 22.04 and 24.04 and openEuler 22.03 and 24.03, with Python 3.10, 3.11 and 3.12 combinations.

Two details of the image matter for the newer agent benchmarks. Since 2026-07-20 the images ship with Docker Engine and Docker Compose v2 preinstalled, because agent style evaluations need to start sibling containers for their sub-tasks. And the container registry publishes the benchmark image as a package, so you can pull it without building anything.

The Python packaging is more than a wrapper script. The setup.py in the repository installs a custom command that downloads the NLTK punkt tokenizer at install time, which is what lets the text processing work without a separate preparation step.

from setuptools import find_packages, setup
from setuptools.command.install import install

The same file parses the requirements files at build time rather than hard coding a dependency list, and it handles both plain requirement files and editable local paths.

Variable length datasets and prefix cache stress testing

The most distinctive capability in the changelog is the variable length and prefix cache workload, added on 2026-09-03, and it is the feature that most affects how a serving team would use the tool.

The dataset generation supports five length modes: fixed lengths, explicit per request lengths, a range, a truncated normal distribution, and a CSV of lengths. Each of these exists because a real serving workload has a shape, and the point of the tool is to reproduce that shape rather than to send uniform prompts.

On top of the dataset, the pipeline covers request scheduling, cache warm-up, metric collection and result analysis as one chain. It supports a single prefix or multiple prefix groups, can back-calculate the parameters needed to hit a target hit rate, distinguishes cold and warmup modes, and supports targeted routing across multiple deployment endpoints. The metrics include prefix_hit_rate, which is queryable from the results.

Two related dataset additions sit alongside it: MRCR 1M, added on 2026-09-02, for evaluating retrieval and question answering over one million token contexts, and corpusQA 1M, added on 2026-08-27, for long sequence reasoning accuracy. Both have their own README under the dataset configuration directory.

The multimodal variant arrived on 2026-09-22, generating variable length multimodal stress datasets from MMMU images, with either a single 1080P image or five 720P images, sharing the same scenario configuration and command entry point as the text only mode.

Agent evaluation through the Harbor framework

Agent evaluation is the area where the harness stopped being an accuracy tool and became an infrastructure tool.

On 2026-09-12 the announcement was that agent evaluation is fully integrated with the Harbor framework, that it supports agents adapted for Harbor and datasets in Harbor format, and that there is a one click offline evaluation deployment for common agents including claude-code, terminus-2 and mini-swe-agent, together with common agent datasets such as SWE-Bench, terminal-bench and DeepSWE.

The benchmark coverage grew over the preceding months. SWE-Bench was the first agent benchmark, added on 2026-04-10, followed by tau squared-Bench on 2026-04-14 for dialogue, tool calling and compliance in a dual control environment. On 2026-05-18 the tool added mini subsets for SWE-Bench, TAU2-Bench and VBench 1.0 to cut evaluation cost. Terminal-Bench 2.0 through Harbor arrived on 2026-05-30, and SWE-Bench Pro for long horizon software engineering tasks on x86 on 2026-06-11.

The mini subsets are the part worth understanding. A full SWE-Bench run is expensive enough in time and tokens that many teams only want it at release boundaries, so a smaller subset gives you a cheap signal for the rest of the cycle. The README notes that the examples for each benchmark are in the corresponding evaluation documentation, where you search for the keyword mini.

The offline deployment path also matters for reproducibility. An evaluation that calls a hosted agent service produces different results depending on when it ran, and the offline option removes that variable.

Judging outputs: judge models and anomaly detection

Evaluation is not only about running a dataset and counting matches, and AISBench covers the two awkward parts of that.

Judge model support was added on 2026-03-01, letting the harness use a separate model to score outputs instead of string comparison. The image generation side goes further: the OneIG-Benchmark integration, added on 2026-06-29, evaluates text to image output across alignment, text rendering, reasoning, style and diversity, in English and Chinese modes, using a hybrid of LLM as Judge and a dedicated small model. The README states that the accuracy difference against the official evaluation method is under one percent, which is the kind of claim worth verifying yourself on your own data before relying on it.

Response anomaly detection, added on 2026-08-19, watches the generated text during an inference evaluation and flags four categories of problem: repetition, garbled characters, rare character output and NaN. It is switched on with a single command line flag, requires no configuration, and writes the findings to JSONL per dataset so they can be audited separately. The important design point is that it does not alter the accuracy or performance metrics; it is a parallel observation of the same run.

The image and video benchmarks sit in the same family. GEdit-Bench, added on 2026-03-10, covers image generation, and VBench 1.0, added on 2026-05-07, evaluates generated video quality and semantic metrics on GPU or NPU.

Performance evaluation beyond accuracy

The 2025-12-19 architecture refactor touched the cli, models, inferencer and tasks components, and the stated goal was to make it quicker to add a new benchmark. That refactor is the foundation under the performance features that followed.

Speculative decoding evaluation, added on 2026-08-10, brings metrics collection, computation and display into the existing performance evaluation flow, so the accepted-token rate and its effect on latency can be measured with the same harness as the rest of the performance run.

Multi LoRA evaluation, added on 2026-07-27, covers the production case where one inference service loads several LoRA adapters for different tenants. The point is that you no longer have to split the evaluation into one run per adapter, which is how multi tenant deployments are actually shaped.

The Mooncake Trace dataset, added on 2026-01-31, brings a trace workload with request scheduling by timestamp, hash_id caching and reproducible prompt generation, which is the right tool when you want to replay the shape of a real request stream rather than a synthetic one.

Taken together, these features point at a specific audience: teams running inference services that need to answer questions about capacity, cache behaviour and tenant isolation, not just teams who want a single accuracy score. That is also where the tool becomes heavy, since each of these workloads wants its own dataset, its own scheduler settings and its own metric definitions.

Extensions, plugins and the licensing picture

The repository layout shows how the project expects to grow. There is an ais_bench directory for the package itself, a plugins directory, a plugin_examples directory, a tools directory, a docker directory with the image overview, a docs directory with the documentation sources, plus smoke_tests, tests and a requirements directory with the split requirement files that setup.py reads.

The plugin structure is the extension point for adding a new benchmark, which matches the stated goal of the refactor. The docs are published on readthedocs, with a Chinese language tree that includes a developer guide for contributors, and the official site is listed separately from the documentation.

The releases are dated master snapshots: v3.1-20261013-master published on 2026-10-16, v3.1-20260912-master on 2026-09-14 and v3.1-20260630-master on 2026-06-30. The last push was on 2026-10-21. The naming tells you the versioning convention, and it also tells you that pinning a release pins a specific snapshot of a moving major version rather than a stable API surface.

On the licence, the README carries an Apache 2.0 badge, while the licence field in the repository metadata is not one GitHub can classify, and the repository root contains a LICENSE file, a NOTICE file and a third party open source software notice. Which terms apply to your use is something to check against the LICENSE file directly and, if you need it, to take to your own legal review.

Editorial conclusion

AISBench suits a team that already runs OpenCompass and now has to evaluate models served over an API, agent systems on software engineering benchmarks, or inference stacks with prefix caching, because it keeps the configuration and dataset conventions of OpenCompass instead of asking you to learn a second harness. It is the wrong choice for a small project that only needs a single accuracy number on one dataset, and it is the wrong choice if you cannot run a container or install the full dependency set. Verify first which benchmark you actually need and whether the mini subsets cover it, read the README of the dataset you plan to use under ais_bench/benchmark/configs/datasets, and confirm the container image architecture matches your host before you plan a multi-node run.

Frequently asked questions

Official sources

1. AISBench/benchmark on GitHub
2. Issues
3. README
4. Releases