OpenBMB/UltraEval-Audio: a unified benchmark harness for ASR, TTS, audio codecs and audio LLMs
Your faithful, impartial partner for audio evaluation — know yourself, know your rivals. 真实评测，知己知彼。A unified benchmark framework for ASR/TTS/Audio Codec/audio LLM evaluation
At a glance
- What is it?
- UltraEval-Audio bundles 34 audio benchmarks, dataset download and metric bindings behind one Python command. It is built for teams that need to reproduce published audio model numbers, not for quick one-off scoring.
- Who is it for?
- Adopt UltraEval-Audio if you need to reproduce published ASR, TTS or audio LLM results and can accept a heavy dependency set and per-model isolated runtimes. Do not adopt it if you only need to score a handful of clips with WER, since jiwer or sacrebleu alone will be faster to set up.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 24 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap UltraEval-Audio fills: audio models evaluated on incompatible harnesses
Audio model papers rarely share an evaluation stack. A TTS paper reports Seed-TTS-Eval numbers with its own text normalisation, an ASR paper reports WER after its own punctuation stripping, and an audio LLM paper reports a Gemini-judged score with a prompt nobody else has. Comparing the three is guesswork.
UltraEval-Audio's answer is to bind datasets and metrics together inside one repository. The README describes it as a framework that aggregates 34 benchmarks across speech, sound, medicine and music, spanning 10 languages and 12 task categories, and that supports both speech understanding and speech generation. The binding is explicit: dataset definitions live under registry/dataset as YAML files, model definitions under registry/model, and the README states that the project binds datasets with official evaluation methods such as WER, WER-ZH, BLEU and G-Eval so that the dataset and the metric stay aligned.
The intended user is not someone who wants a quick score for a demo. It is a team that needs to publish or reproduce a number that another team can regenerate. The replication/ directory is the clearest signal of that intent: it holds per-model documents such as replication/CosyVoice2.md, replication/qwen3_asr.md and replication/FireRedTTS3.md, each described as carrying replication results and one-click commands.
The trade-off is scope. A framework that ships 34 benchmarks and a dozen model integrations carries the dependency weight of all of them, and the requirements.txt file reflects that.
How the registry, isolated runtime and IPC mechanism fit together
The architecture is a registry plus a dispatcher plus a sandboxed worker.
Datasets and models are declared, not hard-coded. A dataset entry under registry/dataset describes where the data comes from and which metric applies; a model entry under registry/model describes how to reach the model. The README points at registry/dataset/long-tts-eval.yaml and registry/model/mgm_omni.yaml as concrete examples. Evaluation starts from audio_evals/main.py, which reads those declarations.
The interesting part is the isolated runtime introduced in v1.1. The release notes state that model-specific dependencies are installed and managed automatically, that inference runs in the isolated environment, and that the isolated process communicates with the main evaluation process over IPC. That design exists for a real reason: an ASR model pinned to one transformers version and a TTS model pinned to another cannot share a process. The changelog shows this in practice, describing Fun-ASR-Nano-2512 evaluation as running through a pinned isolated FunASR runtime.
The cost is operational. Every isolated environment has to be built, and the first run of a new model pays that installation cost before any audio is scored. The README also notes that the project uses uv to accelerate model dependency installation, which is an acknowledgement that installation time is a real friction point.
For throughput, the changelog documents a model pool: adding --use_model_pool together with --workers <N> enables multi-GPU parallel inference. Resume support is handled by -r/--resume, introduced in the 2025/05/08 entry for faster restart after a partial run.
Installing UltraEval-Audio and running a first evaluation
The package metadata declares Python 3.10 or newer and an Apache-2.0 licence, and it exposes a console script named ultraeval-audio. The primary documented entrypoint, however, is the module path, so the commands below follow the README and the changelog rather than the console script.
Start by installing the dependencies from the repository root. The dependency list is declared dynamically from requirements.txt, so a plain install pulls the full set, including datasets, jiwer, sacrebleu, librosa and openai.
pip install -r requirements.txtEvaluations are launched through audio_evals/main.py with a dataset name and a model name. The README gives this shape for the general case, and the changelog uses the same pattern in its GPU parallel example.
python audio_evals/main.py --dataset <dataset_name> --model <model_name>Replace both placeholders with identifiers that exist in the registry. The dataset names and model names are the ones used in registry/dataset and registry/model, and the changelog lists concrete model identifiers such as qwen3-asr-1.7b, voxcpm2, moss-tts-v1.5 and fireredtts3-base.
To use more than one GPU, add the model pool flags. The changelog shows a four-worker example.
python audio_evals/main.py --dataset <dataset_name> --model <model_name> --use_model_pool --workers 4If a run is interrupted, the -r/--resume parameter restarts from where it stopped rather than rescoring everything. What you should see on a first run is the dataset being acquired and prepared (the README states that datasets such as Librispeech, TED-LIUM and Seed-TTS-Eval are downloaded and processed automatically), followed by inference and then the metric output. If you are evaluating a model that has a replication document, read that document first: it carries the exact command and the expected result.
Where UltraEval-Audio gets in your way
The dependency surface is the first obstacle. requirements.txt pins datasets to 3.6.0, jiwer to 2.6.0, sacrebleu to 1.5.1 and websockets to 12.0, and pulls in azure-core, dashscope, modelscope and gdown. Those pins exist to keep evaluation reproducible, but they also mean UltraEval-Audio does not slot cleanly into an existing environment that already has different versions of the same packages. The isolated runtime solves this for model inference, not for the main process.
Automatic dataset download is a second constraint. It is convenient when the download works and a hard blocker when it does not, because a network-restricted cluster cannot fetch Librispeech or TED-LIUM on demand. The README does not document an offline dataset placement procedure, so if your evaluation machines have no outbound access you should treat that as unverified until you check the dataset YAML files yourself.
Coverage is uneven by design. The changelog is dominated by TTS and ASR additions, with audio codec support arriving as part of the v1.1 specialised model work. If your model is a general audio understanding model that is not in registry/model, you will be writing a YAML entry and possibly a metric binding before you can score anything.
Finally, this is the wrong tool for small jobs. If you have 200 clips and want a WER number, installing a framework with 34 benchmarks and an IPC sandbox is a poor trade. jiwer or sacrebleu directly will finish sooner.
UltraEval-Audio compared with running jiwer, sacrebleu and a TTS metric yourself
The realistic alternative is not another framework. It is a script that loads your audio, calls your model, and computes metrics with the underlying libraries directly. UltraEval-Audio depends on several of those libraries anyway, so the comparison is about what the framework adds on top.
That addition is the registry and the replication documents. If you write your own script, the definition of the test set, the text normalisation and the metric configuration live in your code, and a reader has to trust your description of them. In UltraEval-Audio they live in YAML files under registry/, which a reader can inspect, and the replication/ documents state the command and the result so the run can be repeated.
A second difference is the isolated runtime. A hand-written script that evaluates one model does not need process isolation. A harness that evaluates Qwen3-ASR, VoxCPM2 and MOSS-TTS-v1.5 in the same session does, because their dependencies conflict. That is the specific problem the IPC design addresses, and it is not something a single-purpose script has to solve.
What you give up is control. With your own script you choose the metric implementation and can patch it in minutes. With UltraEval-Audio you either use the bound metric or extend the registry. For published comparisons the binding is the point; for exploratory work it is overhead.
Maintenance, licensing and what the repository does not promise
The last push to the default branch was on 2026-08-25, and the repository is not archived. The changelog shows a steady cadence through 2026, with entries for FireRedTTS3, Fun-ASR-Nano-2512, MOSS-TTS-v1.5, VoxCPM2, InstructTTSEval and Qwen3-ASR between April and August 2026. The most recent tagged release listed is v1.1 from 2026-01-06, so if you depend on versioned releases rather than the main branch, note that the changelog has moved well past that tag.
The upgrade cost is concentrated in two places. Model integrations pull their own dependencies into isolated environments, so adding a model can mean a fresh environment build and a new set of pins. Dataset and metric changes can invalidate stored results, because a metric binding change alters the number a run produces. If you keep historical scores, record the commit you ran them at.
The licence is Apache-2.0, declared both in the LICENSE file and in the pyproject.toml classifier. That is permissive and generally friendly to commercial use, but it covers this repository only. The benchmarks it downloads and the models it evaluates carry their own licences, and some speech datasets restrict commercial use. Checking those terms is your responsibility, not something the framework can resolve for you, and this is not legal advice.
Editorial conclusion
Adopt UltraEval-Audio if you need to reproduce published ASR, TTS or audio LLM results and can accept a heavy dependency set and per-model isolated runtimes. Do not adopt it if you only need to score a handful of clips with WER, since jiwer or sacrebleu alone will be faster to set up. Before committing, verify that the registry contains a YAML entry for your exact model, that the dataset you need is listed in registry/dataset, and that your Python is 3.10 or newer.
Frequently asked questions
How do I evaluate audio quality with UltraEval-Audio?
The repository includes a document on using UTMOS and DNSMOS for speech quality evaluation, added in the 2025/05/22 changelog entry. Beyond that, quality scoring depends on the dataset and metric binding you select from the registry, since the framework binds datasets to official metrics such as WER, BLEU and G-Eval.
What is the UltraEval-Audio framework?
It is an open-source evaluation framework for audio foundation models that supports both speech understanding and speech generation. The README states it aggregates 34 benchmarks across speech, sound, medicine and music, covering 10 languages and 12 task categories.
Which Python version does UltraEval-Audio require?
The pyproject.toml declares requires-python as >=3.10, and the package classifiers list Python 3 only. The console script ultraeval-audio is also declared there, pointing at audio_evals.main:main.
How do I run UltraEval-Audio across multiple GPUs?
The changelog states that adding --use_model_pool together with --workers <N> enables multi-GPU parallel inference, and gives the example of --workers 4. The same entry shows the flags appended to the standard python audio_evals/main.py invocation.
Community notes