UltraEval-Audio: a manifest versioned 0.0.0 beside a v1.1 tag, and a registry the wheel cannot carry
Your faithful, impartial partner for audio evaluation - know yourself, know your rivals. A unified benchmark framework for ASR/TTS/Audio Codec/audio LLM evaluation.
At a glance
- What is it?
- A harness that binds 34 speech, sound, medical and music benchmarks to their metrics and runs models in isolated per model environments. Its build only packages the audio_evals tree, so the registry YAML and the replication notes the README tells you to read are not in the installed wheel at all.
- Who is it for?
- UltraEval-Audio is the right shape for anyone who has to score the same audio models against the same metrics and cannot afford to rewire datasets to metrics by hand, and it is the wrong shape for anyone who installs from PyPI and expects the shipped configuration to arrive. Run it from a checkout rather than a wheel, because the registry and the replication notes are outside the packaged tree.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 12 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- OpenBMB/UltraEval-Audio329 ★ · +5
| Repository | 2026-09-21 | 2026-11-15 |
|---|---|---|
| OpenBMB/UltraEval-Audio | 324 | 329 |
5 Stars in the last 55 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The registry YAML the README points at is not inside the packaged tree
The packaging configuration is narrow. `[tool.setuptools.packages.find]` sets `where = ["."]` and `include = ["audio_evals*"]`, so the only Python package installed is the `audio_evals` tree. The exclude list adds `tests*`, `env*`, `envs*`, `raw*`, `outputs*`, `third_party*` and `replication*`.
Package data is scoped the same way, `[tool.setuptools.package-data]` setting yaml, txt, json and md globs against `audio_evals` only. So YAML files are shipped if they sit inside that package and not otherwise.
Now look at what the README sends you to. Two changelog entries point at files at the repository root: `registry/model/seed_audio.yaml` for the Seed Audio API configuration, and `registry/dataset/long-tts-eval.yaml` for dataset alignment. Every replication entry points into `replication/`, a directory the exclude list names explicitly, and the repository holds `registry/`, `cli/`, `docs/`, `paper/` and `assets/` at the root as well.
None of those paths are inside `audio_evals*`. A wheel built from this configuration contains the harness and the configuration that happens to live inside it, and not the registry it reads, the model notes it cites, or the dataset alignment files that changelog entries tell you to consult. The console script `ultraeval-audio = "audio_evals.main:main"` will therefore start in an installation that is missing most of what the documentation is about.
The manifest reports 0.0.0 and the only release is v1.1
`pyproject.toml` writes `version = "0.0.0"` as a literal, and it is not on the dynamic list. The dynamic declaration covers dependencies only, pointing at `requirements.txt`.
So any build that does not substitute a version produces a distribution reporting itself as 0.0.0. The single GitHub release is tagged v1.1 and dated 2026-01-06, and the changelog entry that announces the release is dated 2025-12-31. Two dates for one version, six days apart.
Nothing in the metadata ties the two together. There is no `[tool.setuptools_scm]` section, no version file read by the build, and no git based backend in the build requirements, which are `setuptools>=68` and `wheel`. The build system is deliberately static, and the static value is the placeholder.
The rest of the metadata is sparse in the same way. There is one author, `UltraEval Contributors`, no maintainer, a homepage pointing at the GitHub repository, and an issues URL. The repository record lists no separate homepage beyond that. `requires-python` is `>=3.10` with no ceiling.
The build requirement for setuptools is also the only floor in the whole file. Everything else about the build is unbounded, including `wheel`, which has no constraint at all.
Twenty six of the thirty one dependencies have no version at all
The dependency list is read straight out of `requirements.txt`, and it is thirty one entries long. Five carry a version. `datasets==3.6.0`, `jiwer==2.6.0`, `sacrebleu==1.5.1` and `websockets==12.0` are exact, and `openai>=1.0.0` is the only floor.
`websockets==12.0` is the outlier worth noting. It is an exact pin on a release from well before the project itself was published, sitting in the same file as a float pin on `datasets` and a caret on `openai`. Three different pinning styles in five lines, with no comment explaining any of them.
The metric libraries are in there too, which is how the binding of datasets to metrics actually happens: `jiwer` for error rates, `sacrebleu` for BLEU, `pycocoevalcap` for the caption metrics, and `editdistance` for raw edit distance. Chinese handling has three entries, `zhconv`, `zhon` and `langdetect`, which is consistent with the advertised `WER-ZH` metric and with a framework that claims ten languages, though no list of those ten appears in the file.
Two entries do not belong to a runtime install at all. `pre-commit` is a development tool, and a `.pre-commit-config.yaml` sits at the root, so a hook installer is a hard dependency of every user. `pytest` is likewise a declared dependency rather than a development extra, while `tests/` is a top level directory.
There is a misspelled directory at the root, `requirments/`, beside the correctly spelled `requirements.txt`, and the build configuration carries a comment noting that the filenames keep the project's current spelling.
Two dependency systems, one for the harness and one per model
The v1.1 highlights describe an isolated inference mechanism. Model specific dependencies are installed and managed automatically, inference runs in the isolated environment, and the two processes talk over IPC, which the file credits with eliminating dependency conflicts.
That mechanism is the answer to a version management problem the requirements file cannot solve. A harness that scores ASR, TTS and audio codec models cannot hold one version of every modelling stack in one environment, so the per model runtime is where the versions live. A later entry describes exactly that for one model, calling it a pinned isolated FunASR runtime for Fun-ASR-Nano-2512.
The older changelog records the other half of the same story: an entry from 2025-10-30 says uv is used to accelerate model dependency installation. So the per model installation path is uv, invoked by the framework, while the harness's own dependencies come from a file with almost no versions in it.
The consequence is that the isolation guarantee applies to models, not to the harness. A framework whose stated purpose is reproducible evaluation pins four of its own thirty one dependencies and lets the other twenty seven float, then wraps every model in a pinned environment. The reproducibility claim is doing work on one side of the boundary only.
Four command line flags are documented, all of them inside a changelog bullet
The only complete command in the file is one line, nested two levels deep under a February 2026 entry:
python audio_evals/main.py --dataset <dataset_name> --model <model_name> --use_model_pool --workers 4That names four flags. Two of them, `--use_model_pool` and `--workers`, arrived with GPU parallel acceleration and are documented as a sub bullet of that one entry. The other two are the only shape the framework's own interface is ever shown in.
Meanwhile the feature list advertises preview testing, random sampling, error retries and resume from breakpoint, each of which implies at least one more flag, and none of them is given a name anywhere in the file. Neither is the `--seed` that a framework claiming reproducibility would need, nor the judge configuration that the generative metrics require.
The invocation form has its own wrinkle. The documented command runs a script path, `python audio_evals/main.py`, while the metadata declares a console script named `ultraeval-audio` pointing at `audio_evals.main:main`. The comment above that declaration calls the entry point optional and ties it to being installed, editable or as a wheel. So there are two ways in, one documented, and the documented one only works from a checkout.
Ten languages claimed, and individual benchmarks covering twenty two and twenty four
The overview states ten languages and twelve task categories across four domains, speech, sound, medicine and music, aggregating what it calls 34 authoritative benchmarks. Three are named as examples, Librispeech, TED-LIUM and Seed-TTS-Eval.
The changelog then counts languages per benchmark. MiniMax multilingual TTS is described as covering 24 languages in the August 2026 entry for FireRedTTS3, and as covering 22 languages in the July 2026 entry for MOSS-TTS-v1.5 and again in the April 2026 entry for Fish Speech S2 Pro.
So a single benchmark inside the framework covers twenty two or twenty four languages while the framework itself claims ten. Those are different units, language coverage per dataset against language coverage across the suite, and the file does not distinguish them. A reader taking the headline number will understate what one dataset exercises, and a reader taking the per dataset number will have no idea how many distinct languages the whole suite touches.
The task categories are not enumerated either. The metric list is short, WER, WER-ZH, BLEU and G-Eval, against twelve categories, so most categories are scored by metrics that appear nowhere in the overview.
Generative metrics need a judge, and no judge is configured anywhere
G-Eval appears in the metric list as one of the methods bound to datasets. One changelog entry, for InstructTTSEval, is explicit about what that costs. It evaluates fine grained acoustic control, descriptive style following, and role play or scenario style following, and it uses a Gemini judge.
So at least one metric in the suite is an opinion from another model, and the framework needs a Gemini endpoint to produce it. The only environment variable named anywhere in the file is `SEED_AUDIO_API_KEY`, set before running so the Seed Audio TTS models can be reached.
Nothing states which Gemini model the judge uses, what prompt it is given, how many samples are drawn, how the scores are aggregated, or whether the judge's output is written into the results directory in a form that can be re-read. Nothing states whether the judge can be swapped for a local model, or what happens to a run where the judge call fails.
That matters for a framework whose stated purpose is impartial evaluation with published replication reports. A judge based score is only comparable to another judge based score if the judge is the same, and the one setting that would make it reproducible is not written down.
The header closes a heading with two unclosed paragraphs, and one doc filename encodes its own spaces
The README opens with a level four heading element, immediately nests a paragraph inside it, then nests a second paragraph inside that one. Paragraph elements cannot nest, and the block is closed with a closing heading tag, leaving both paragraphs unterminated. The language switcher, the Discord link and the paper link all sit inside that structure, so the top of the file renders differently depending on the parser.
The paper link points at an ACL Anthology entry whose identifier marks it as a demo, `2026.acl-demo.56`, and the announcement above it says the work was published at ACL 2026. There is no citation block: no authors, no year, no venue string, no BibTeX. The repository does carry a `paper/` directory, which the README never mentions.
The changelog lives inside the README rather than in a changelog file, and the repository root has no changelog document. It does have an `FAQ.md`, also unlinked from the README.
One documentation path encodes its own punctuation. The entry for using audio quality metrics links to `docs/how%20use%20UTMOS%2C%20DNSMOS%20eval%20speech%20quality.md`, a filename built from percent encoded spaces and an encoded comma. It is reachable through the link and awkward to type, to quote, or to reference from a shell.
The suite's own superlative is also worth reading carefully. The overview calls this the world's first open source framework supporting both speech understanding and speech generation evaluation, and promises unprecedented convenience and efficiency. Neither claim carries a date, a comparison set, or a citation.
Editorial conclusion
UltraEval-Audio is the right shape for anyone who has to score the same audio models against the same metrics and cannot afford to rewire datasets to metrics by hand, and it is the wrong shape for anyone who installs from PyPI and expects the shipped configuration to arrive. Run it from a checkout rather than a wheel, because the registry and the replication notes are outside the packaged tree. Before you rely on a number, pin the framework dependencies yourself, since 26 of the 31 entries carry no version at all, and check which judge model the generative metrics use, since no judge configuration is documented anywhere in the file.
Frequently asked questions
How do I run an UltraEval-Audio evaluation?
With `python audio_evals/main.py --dataset <dataset_name> --model <model_name>`, adding `--use_model_pool` and `--workers <N>` for GPU parallel inference. The metadata also declares a console script named `ultraeval-audio` pointing at `audio_evals.main:main`, which the documentation never shows. Feature flags for preview testing, sampling, retries and resume are advertised without being named.
What is the current version of UltraEval-Audio?
The only GitHub release is v1.1, dated 2026-01-06, while the changelog announces v1.1 on 2025-12-31. `pyproject.toml` writes `version = "0.0.0"` as a literal rather than deriving it from a tag, so a local build reports 0.0.0.
How do I evaluate audio quality with UltraEval-Audio?
The repository ships a guide on using UTMOS and DNSMOS for speech quality, reachable from a path under `docs/` whose filename is percent encoded. The metric stack also includes `sacrebleu==1.5.1` for BLEU, `jiwer==2.6.0` for error rates and `pycocoevalcap` for caption metrics, bound to datasets by the framework itself.
Does UltraEval-Audio need API keys to run?
For the hosted models, yes. The changelog names one variable, `SEED_AUDIO_API_KEY`, set before running so the Seed Audio TTS models can be reached. The generative metrics also rely on a Gemini judge for InstructTTSEval, and no judge configuration, model name or key is documented for it.
Which languages does UltraEval-Audio cover?
The overview claims ten languages and twelve task categories over four domains, with 34 benchmarks. Individual changelog entries name benchmarks covering 22 languages and, for one model, 24, so the per dataset count and the suite count are different units and neither is enumerated.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/openbmb-ultraeval-audio)Star history chart for your README
[](https://hysenlabs.com/projects/openbmb-ultraeval-audio)Was this useful?