MiMo-Audio: a 7B audio language model built for few-shot audio work

Why MiMo-Audio treats audio as a language modelling problem

The premise is stated plainly in the README. Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks, while people generalise to a new audio task from a few examples or a simple instruction. The GPT-3 argument is then transferred to audio: scaling next-token prediction pretraining should produce the same generalisation in the audio domain, and the authors state that they believe this paradigm is equally applicable to audio.

The claim is backed by a pretraining scale figure. The README says the pretraining data was scaled to over one hundred million hours, and that few-shot learning capabilities emerged across a diverse set of audio tasks as a result. That emergence claim is the interesting part of the release, because it is a claim about behaviour that was absent from pretraining rather than about a benchmark score.

The README further states that MiMo-Audio-7B-Base achieves state of the art performance among open source models on speech intelligence and audio understanding benchmarks, and that the model generalises to tasks absent from its training data, such as voice conversion, style transfer and speech editing. It also describes speech continuation capability that produces talk shows, recitations, livestreaming and debates. These are the project authors' own claims from the README and the technical report, not independent verification.

For a reader, the audience is clear. This is for engineers who want a single audio model that handles understanding and generation, and who would rather prompt a model with examples than assemble a separate fine-tuning pipeline for every new task.

The tokenizer, the patch encoder and the 6.25 Hz bridge to the LLM

MiMo-Audio-Tokenizer is described as a 1.2B parameter Transformer operating at 25 Hz, with an eight layer RVQ stack that produces 200 tokens per second. The README says it is trained from scratch on a ten million hour corpus with semantic and reconstruction objectives optimised jointly, and that this yields better reconstruction quality while helping downstream language modelling. Residual vector quantisation gives a discrete vocabulary, which is what makes next-token prediction possible over audio at all.

The full model couples a patch encoder, an LLM and a patch decoder, and the README gives the reason: this arrangement improves modelling efficiency for high rate sequences and bridges the length mismatch between speech and text.

The arithmetic is worth following because it explains the design. The patch encoder aggregates four consecutive time steps of RVQ tokens into a single patch, which downsamples the sequence to a 6.25 Hz representation before the LLM sees it. The patch decoder then reverses this by autoregressively generating the full 25 Hz RVQ token sequence through a delayed generation scheme. A 200 tokens per second stream becomes 50 tokens per second at the LLM, and the LLM's job stays closer to a text model's job, which is where the generalisation argument comes from.

The cost is a two-stage audio path. Any pipeline that wants to manipulate, align or compare audio has to reason about both the 25 Hz token space and the 6.25 Hz patch space.

Base versus Instruct: the difference in how you drive the model

Two checkpoints are published and they are not interchangeable. MiMo-Audio-7B-Base is the pretrained model, and the repository ships inference_example_pretrain.py to explore what the README calls in-context learning capabilities. MiMo-Audio-7B-Instruct is the post-trained model, driven by inference_example_sft.py.

The post-training stage is described in the README: a diverse instruction-tuning corpus was curated, and thinking mechanisms were introduced into both audio understanding and generation. The README reports that MiMo-Audio-7B-Instruct reaches open source state of the art on audio understanding benchmarks, spoken dialogue benchmarks and instruct-TTS evaluations, approaching or surpassing closed source models.

Practically, the base model is the one to reach for when your task has no instruction template, for example a speaker or emotion style you want to capture from example clips and reuse. The instruct model is the one to reach for when a user will type a request and expect a spoken answer, or when you need a controllable text to speech path.

The example directory gives a hint at the intended input formats. It contains male and female English speech prompts, a male and female Chinese speech prompt, a spoken dialogue assistant turn, and a folder named ESD, which corresponds to the speech editing and voice conversion capability the README lists. The English and Chinese prompt pairs are the ones to use for testing the model's handling of voice identity.

Downloading the checkpoints from Hugging Face

The three model files are distributed through the Hugging Face Hub under the XiaomiMiMo organisation. The tokenizer, the base model and the instruct model each have their own repository, and the README lists them in a download table.

The download path uses the huggingface-hub command line tool, and the local directory layout below the models folder is what the demo expects later.

pip install huggingface-hub

hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir ./models/MiMo-Audio-Tokenizer
hf download XiaomiMiMo/MiMo-Audio-7B-Base --local-dir ./models/MiMo-Audio-7B-Base
hf download XiaomiMiMo/MiMo-Audio-7B-Instruct --local-dir ./models/MiMo-Audio-7B-Instruct

If you only plan to run the instruct demo you can fetch the tokenizer and the instruct checkpoint and skip the base model, which is the largest of the three downloads.

The repository also points to a browser based demo on the Hugging Face Spaces platform, so a first look at the model does not require the local install at all. The local Gradio app described below is for people who want the checkpoints on their own hardware.

Running the local Gradio demo on Linux with CUDA 12

The README states the prerequisites for the local setup: Python 3.12 and CUDA 12.0 or later, listed for Linux. The installation sequence clones the repository and installs the pinned requirements, then adds flash-attn at a fixed version.

git clone https://github.com/XiaomiMiMo/MiMo-Audio.git
cd MiMo-Audio
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1

The requirements file pins torch and torchaudio at 2.6.0, transformers at 4.49.0 and gradio at 5.46.1, along with triton 3.2.0, librosa, fastapi, uvicorn and pydantic. The pin on torch 2.6.0 matters more than usual here, because the flash-attn wheel offered by the project is built for cu12 and torch 2.6.

The README warns that compiling flash-attn can take a long time, and gives a precompiled wheel for Python 3.12 on Linux x86_64 that you can install directly instead.

pip install /path/to/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl

With that in place, the demo starts with a single command.

python run_mimo_audio.py

That launches a local Gradio interface. The README tells you to enter the local paths for MiMo-Audio-Tokenizer and MiMo-Audio-7B-Instruct in the interface, so the paths you used for the hf download step are the values to type in.

The separate inference scripts are the route for scripted work rather than interactive use, and the README points to them by filename only, so what arguments they accept has to be read from the files themselves.

Where MiMo-Audio is the wrong choice

The first constraint is hardware. The prerequisites are Linux, Python 3.12 and CUDA 12.0 or later, and the dependency set includes a pinned torch 2.6.0 with triton 3.2.0 and a flash-attn build. There is no CPU path, no macOS path and no Windows path described in the README. A 7B model plus a 1.2B parameter tokenizer, with a decoder that expands back to 25 Hz tokens, is a GPU workload.

The second constraint is scope. This is a speech and general audio language model, not an ASR system. The README lists speech intelligence, audio understanding, instruct-TTS, spoken dialogue, voice conversion, style transfer and speech editing, and says nothing about transcription accuracy or timestamps. If the task is a transcript, this is the wrong repository.

The third constraint is the nature of the few-shot claim. The README describes generalisation to tasks absent from the training data, and that is a behaviour you have to confirm on your own data distribution. Few-shot performance is sensitive to the quality and diversity of the examples you provide, and the repository gives you a script rather than a recipe. There is no documented guidance in the README on how many examples, or how to select them.

The fourth constraint is the release state. The repository has no GitHub releases, and the last push was on 2026-06-17. Checkpoints exist on the Hub and the code is on the main branch, but there is no tagged version to pin, so a reproducible environment depends on the commit you clone rather than a release artefact.

MiMo-Audio next to Qwen2-Audio and the AudioLM family

The comparison worth making is with other open audio language models that expose a similar prompt interface, and the difference in approach is instructive.

Qwen2-Audio takes the route of a single general model with audio input encoded in place of text tokens, trained on a large mixture of audio-text pairs and evaluated across a broad set of speech and audio benchmarks. It is convenient because one checkpoint handles question answering over audio, captioning and speech recognition, and it ships with an instruct tuned variant for the same reason MiMo-Audio does.

MiMo-Audio differs in two specific ways. The first is the audio front end. Rather than encoding a spectrogram patch directly into the language model, it uses a separate 1.2B parameter RVQ tokenizer trained for reconstruction and semantic quality, and then a patch encoder that downsamples that 200 tokens per second stream to 6.25 Hz. The consequence is that the LLM operates on a tokenised, quantised audio vocabulary, and the same machinery can generate audio as well as consume it.

The second difference is the emphasis on the base model. The README treats MiMo-Audio-7B-Base as the centre of the release and positions few-shot learning as the outcome of pretraining scale, with the instruct model as a post-trained layer on top. AudioLM and the AudioPaLM line of work established the text and speech token interleaving idea, and their tooling and evaluation ecosystems are older and broader.

A team choosing between them is really choosing between a tokenizer-first design that can generate and edit audio, and a spectrogram-input design with a shorter path from audio to language.

Licence, evaluation tooling and the cost of adopting the weights

The repository is licensed under Apache 2.0, with the licence text in the LICENSE file. The citation block in the README attributes the work to LLM-Core-Team Xiaomi, and the technical report is published in the repository as MiMo-Audio-Technical-Report.pdf. A contact address, [email protected], is given alongside opening an issue for questions.

The evaluation code is a separate repository, MiMo-Audio-Eval, and the README describes it as a flexible and extensible framework for evaluating MiMo-Audio and other recent audio LLMs, supporting a wide range of datasets, tasks and models. That separation is worth noting for anyone planning to publish their own numbers, since the harness is not vendored into the main repository.

The adoption cost is mostly storage and evaluation time. Three checkpoints have to be downloaded, the environment has to match the pinned torch and flash-attn versions, and the claims in the README are claims about generalisation, which can only be reproduced on your own data with the eval harness or a script of your own. The deployment story is thin by design: the README covers the demo, the two inference scripts and the eval toolkit, and says nothing about serving, quantisation, throughput or latency. Plan for a research and prototyping deployment rather than a production service, and check the transformers 4.49.0 pin in requirements.txt before planning any of that work.

Editorial conclusion

Adopt MiMo-Audio if you need an Apache 2.0 audio model that handles speech understanding, instruct TTS and spoken dialogue from one checkpoint on a single CUDA 12 machine, and if you have examples of your target task to prompt with. Skip it if your work is offline transcription, if you cannot meet the Python 3.12 and flash-attn install path, or if you need a tokenizer that runs outside the 25 Hz RVQ scheme. Before committing, run the base model on examples/prompt_speech_en_m.wav and check whether the reported few-shot results hold on your own audio distribution.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. XiaomiMiMo/MiMo-Audio on GitHub