UTMOSv2: predicting speech naturalness MOS with a pretrained model and an inference script

The problem UTMOSv2 addresses: replacing a listening panel with a scalar

Mean opinion score is the standard way speech synthesis work reports naturalness, and it is normally collected by asking human listeners to rate samples on a fixed scale. That process is slow, expensive, and hard to repeat across model checkpoints. UTMOSv2 exists to produce a predicted MOS instead: a single number per audio file, computed by a model rather than a panel. The repository is the official implementation of a system submitted to the VoiceMOS Challenge 2024, described in the linked paper as transfer learning from a deep image classifier to naturalness MOS prediction of high-quality synthetic speech. The intended user is someone evaluating TTS or voice conversion output at scale, where running a new listening test for every checkpoint is not practical. It is not a general audio quality metric. The topics list says speech-synthesis and speech-recognition, but the README frames the target as naturalness of high-quality synthetic speech, and nothing in the supplied material claims it handles noisy real-world recordings or codec artifacts.

What the pretrained path actually does when you call predict

The public surface is small. You call utmosv2.create_model(pretrained=True), which returns a model object, and then model.predict(...) with one of three input forms. Pass data as a torch.Tensor or np.ndarray shaped (batch_size, sequence_length) or (sequence_length,) and you get back a tensor or array shaped (batch_size,) or (1,). Pass input_path for a single .wav and you get a float. Pass input_dir and you get a list of dicts with file_path and predicted_mos keys. The sampling rate is a parameter: the README states that if sr is not provided it defaults to 16000 Hz, which means the model expects you to tell it the rate of your audio rather than resampling silently. The precedence rules are documented as notes: when data is provided, input_path and input_dir are ignored; when data is None, exactly one of input_path or input_dir must be given, not both. The architecture is not described in the README beyond the paper title, so the image-classifier backbone and the transfer-learning details are only available by reading arXiv 2409.09305. If you need to know what the model is doing internally before trusting its scores, the README alone will not tell you.

Installing from GitHub and running inference.py

Two install routes are documented, and they are not equivalent. For library use, the README gives uv add git+https://github.com/sarulab-speech/UTMOSv2.git, with pip install git+https://github.com/sarulab-speech/UTMOSv2.git as the pip equivalent. That pulls the package from the repository rather than PyPI, so version pinning means pinning a commit or tag. For the script route, you clone the repository, cd into UTMOSv2, and run uv sync --extra optional, or with pip: pip install --upgrade pip followed by pip install -e .[optional]. The README flags a zsh detail: escape the brackets as pip install -e '.[optional]'. Then python inference.py --input_path /path/to/wav/file.wav --out_path /path/to/output/file.csv, or swap in --input_dir for a folder. One practical note is worth repeating: if --out_path is omitted, results go to standard output, which the README says is useful when the number of files is small. The same either/or constraint applies here, either --input_path or --input_dir, not both. Python versions listed on the badge are 3.9 through 3.13.

The accuracy caveat the README puts in a note

The most important sentence in the quick-start is a note, not a feature: these methods provide quick and simple predictions, and for more accurate predictions and detailed usage you should refer to the inference guide at docs/inference.md. That is an admission that the convenient API is not the best-performing configuration. A team that wires model.predict into a CI job and never opens docs/inference.md may be shipping numbers the authors consider suboptimal. The README does not state how large the gap is, what the accurate configuration changes, or which checkpoints or ensembles it uses. That is a real documentation gap, and it is the first thing to resolve before treating any score as a regression threshold. The second gap is scope: the README reports challenge placement but does not publish per-corpus or per-domain error rates, so you cannot tell from the README how the model behaves on, say, heavily accented speech or low-bitrate audio. Verify on your own data before you gate a release on it.

Training your own model and what the repository expects

The README has a section titled Train UTMOSv2 Yourself, but the supplied text is truncated mid-sentence at a link to a training guide, so the concrete commands, config keys and dataset format for training cannot be confirmed from this material. What can be confirmed is that training is a supported path rather than an afterthought, and that the optional dependency extra exists, which suggests the training stack pulls in packages the inference path does not need. Treat this as the weakest part of the README: if your plan is to fine-tune on in-house MOS ratings, budget time to read the training documentation in the repository rather than the landing page. The MIT licence is permissive and imposes no copyleft obligation on your own code, but it says nothing about the provenance or licensing of the pretrained weights or the training corpora, and the supplied material does not address either. Check that separately if you plan to ship the model inside a product.

Where a learned MOS predictor is the wrong instrument

A predicted MOS is a model output, not a measurement. If your decision depends on detecting a specific defect, such as a click, a mispronunciation, or a channel artifact, a single naturalness scalar will average that defect away across the utterance. UTMOSv2 also gives you one number per file, so it cannot tell you which segment of a long clip degraded. The model was built for high-quality synthetic speech, and the README makes no claim about robustness to noisy or far-field audio. Finally, because the model is trained against human ratings, it inherits their biases; if your target listeners differ from the rating population behind the training data, the score may not track your users. In those cases a targeted classifier or a small human panel answers the question better than a MOS regression head. The right role for UTMOSv2 is triage and ranking across many candidates, not final acceptance.

The honest alternative: build or fine-tune your own rater

The realistic alternative is not a different package but a different approach: collect a few hundred of your own ratings and fine-tune a model, which the README's training section exists to support. The difference in method is the training data. A pretrained UTMOSv2 gives you a score calibrated to whatever corpora the authors used, with no visibility from the README into that calibration. A model trained on your own ratings is calibrated to your listeners and your audio conditions, at the cost of annotation effort and the risk that a few hundred ratings is too thin to train on. The trade-off is coverage against fit. Use the pretrained model when you need a number today and your audio resembles the target domain; train when the score has to correlate with your own listeners. The repository supports both, which is more than many research releases do. The VoiceMOS Challenge 2024 result cited in the README is the evidence that the pretrained configuration is competitive under that challenge's conditions, and it is the only performance evidence the supplied material provides.

Editorial conclusion

Adopt UTMOSv2 if you need a pretrained naturalness MOS predictor you can call from Python or from inference.py without building a listening test pipeline, and if you are willing to read docs/inference.md because the README itself states that its quick examples are not the accurate path. Do not adopt it if you need a model whose accuracy you can verify from the README alone, or if you cannot accept that the repository does not publish per-domain limitations. Verify first that your audio matches the 16000 Hz default the API assumes, and that your use case tolerates a single scalar score per file.

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. sarulab-speech/UTMOSv2 on GitHub