AudioBench: a universal benchmark for audio large language models

What AudioBench solves, and who it is for

Evaluating an audio large language model is awkward because the tasks do not share a scoring method. Word error rate works for transcription. BLEU works for speech translation. Multiple-choice question answering needs something that can read a free-form answer and decide whether it is correct. AudioBench puts all three under one harness and reports them through a single evaluation script, so a team comparing two checkpoints does not have to write three separate scoring paths.

The repository describes itself as "a universal benchmark for evaluating audio large language models (AudioLLMs) on speech, audio-scene, and voice understanding tasks across 50+ datasets". The audience is narrow but real: researchers preparing a paper, and engineers deciding which open audio model to fine-tune or deploy. The arXiv paper (2406.16020) is the reference point for the methodology, and the Hugging Face Space hosts the live leaderboard.

If your work never touches audio, this is not your tool. The entire harness assumes audio inputs and audio-conditioned models.

How the evaluation pipeline actually runs

The design splits into two processes. The model under test produces answers. A separate judgement model scores them. The README is explicit that for model-as-judge evaluation the judgement model is served "as a service via vllm on port 5000", and the example uses an int4 quantized Llama-3-70B-Instruct requiring one 80GB GPU. Inference for the model under test runs on a second GPU, which is why the quick start sets GPU=2.

That split is the most consequential architectural choice in the repository. It means a full evaluation needs two GPUs, not one, and the judge must be running before eval.sh is invoked. It also means judge-based metrics are not deterministic in the way wer is: the score depends on which judge model and which quantization you served.

The repository points to ARCHITECTURE.md for how the pieces fit together, and the top-level layout separates concerns: src/ for implementation, scripts/ for helper scripts, examples/ for per-model shell scripts and dataset documentation, log_for_all_models/ for stored results, and leaderboard/ for the ranking side. Datasets and metrics are named strings passed to eval.sh, and the full mapping lives in examples/supported_datasets.md.

Installing AudioBench and running a first evaluation

The README gives one installation step, a pip install against requirements.txt. Run it from the repository root:

pip install -r requirements.txt

requirements.txt carries a warning worth reading before you commit to an environment. The versions are unpinned, and the file suggests freezing exact versions after a successful install. It singles out transformers as the most version-sensitive dependency: Qwen2-Audio requires >= 4.45, but the comment notes that some newer releases have regressed audio model behaviour, and points to issue #10. So the install command is short, but the environment it produces is not reproducible by default.

The quick start then has two steps. First serve the judge model:

bash vllm_model_judge_llama_3_70b.sh

The README states this example uses an int4 quantized Llama-3-70B-Instruct and requires 1 * 80GB GPU. Second, run the evaluation on a different GPU:

GPU=2
BATCH_SIZE=1
OVERWRITE=True
NUMBER_OF_SAMPLES=-1
MODEL_NAME=Qwen2-Audio-7B-Instruct
DATASET=cn_college_listen_mcq_test
METRICS=llama3_70b_judge
bash eval.sh $DATASET $MODEL_NAME $GPU $BATCH_SIZE $OVERWRITE $METRICS $NUMBER_OF_SAMPLES

NUMBER_OF_SAMPLES=-1 means all test samples. To switch tasks, the README says to replace DATASET and METRIC; its own example moves to librispeech_test_clean with the wer metric, which does not need a judge model at all. Expect results to land in the log directory the harness uses, and note that OVERWRITE=True will replace previous runs for the same dataset and model.

Where AudioBench gets in your way

The two-GPU requirement is the first wall. Serving a 70B judge on one 80GB card and running inference on a second is fine in a lab and painful on a single-GPU workstation. The wer and acc metrics avoid the judge entirely, so transcription and the spoken-mqa accuracy tasks are cheaper to run, but the question answering and instruction tasks default to judge scoring.

Environment drift is the second wall. The requirements file deliberately leaves versions open and warns that transformers releases have regressed audio model behaviour. A score you record today may not reproduce on a fresh machine next month unless you freeze versions yourself, as the file instructs.

Third, adding a model is a documented but manual path. The repository ships examples/adding_new_model.md and per-model scripts such as examples/eval_SALMONN_7B.sh, which tells you the integration surface exists, but there is no plugin registry or automatic model discovery described in the README. If your model does not fit the expected inference interface, you are writing adapter code.

Finally, judge-based metrics are not comparable across judge choices. A score produced with llama3_70b_judge and one produced with gpt4o_judge come from different scorers, so mixing them in a comparison table is a mistake the harness will not stop you from making.

Alternatives and how their approach differs

VoiceBench, SD-Eval and MMAU appear in the related searches for this project, and they represent the same broad idea: benchmark suites for spoken and audio-capable models. The difference worth understanding is scope versus depth. AudioBench's stated position is breadth across 50+ datasets spanning ASR, speech translation, speech question answering, speech instruction, audio scene question answering and instruction following, with a leaderboard aggregating results. A narrower benchmark that concentrates on one task family will usually give you a sharper signal on that family and a weaker one everywhere else.

The second axis is scoring. AudioBench mixes objective metrics (wer, bleu, acc) with model-as-judge metrics. A benchmark built only on objective metrics is cheaper to run and easier to reproduce, but it cannot score open-ended spoken answers. If your evaluation budget is one GPU and a few hours, the judge-based half of AudioBench is effectively out of reach, and an objective-metric-only suite is the more practical choice.

Maintenance, licensing and upgrade cost

The repository is not archived, and the last push was on 2026-05-29. There are no releases retrieved, so there is no versioned artifact to pin against; you track the main branch. That changes the upgrade calculus. Pulling the latest commit can change dataset definitions or metric implementations, and without releases there is no changelog boundary to diff against beyond the README's Change Log section. For published results, record the commit hash alongside your scores.

The licence is reported as NOASSERTION, meaning the repository's licence file could not be classified automatically. The top-level entries include a LICENSE file, so a licence exists, but its terms are not summarised here and this is not legal advice. Before using AudioBench in a commercial evaluation pipeline, read that LICENSE file directly and check the terms of the datasets it downloads, since those are separate works with their own conditions.

Editorial conclusion

Adopt AudioBench if you are choosing between audio LLMs and need comparable numbers across ASR, speech translation, speech question answering and audio scene tasks, and you have at least one large GPU for the judge model. Do not adopt it if you need a stable, pinned environment out of the box, or if your model cannot be wired into the repository's inference path, since examples/adding_new_model.md is the only documented route. Before running anything, verify that your transformers version satisfies the >= 4.45 floor in requirements.txt, that your GPU count matches the two-GPU split in the quick start, and that the dataset and metric names you plan to use appear in examples/supported_datasets.md.

Frequently asked questions

Official sources

1. AudioLLMs/AudioBench on GitHub
2. Issues
3. Project website
4. README