MiniMind-O: a 0.1B omni model you can actually train from scratch

What MiniMind-O solves, and who it is for

The common way to build a voice assistant is a cascade: an ASR model transcribes speech to text, an LLM answers in text, and a TTS model speaks the answer. That works, but every hop through text discards prosody and adds latency. MiniMind-O takes the other route. Speech and text meet at the hidden-state level inside one backbone, so the model does not have to write down what it heard before responding. The README frames this as filling a gap: plenty of large omni models exist, but few lightweight ones ship the complete training path. MiniMind-O is the third entry in the MiniMind series after MiniMind (language) and MiniMind-V (vision), and it reuses that series' design conventions. The target reader is a developer or student who wants to train and modify an omni model end to end, not someone shopping for a hosted voice API. The published weights are minimind-3o at roughly 0.1B parameters and minimind-3o-moe at roughly 0.3B total with 0.1B active.

Thinker, Talker and the frozen encoders between them

The architecture splits into two paths. A Thinker handles understanding across text, audio and vision; a separate Talker produces speech. Audio and image inputs are encoded by frozen models before they ever reach the language backbone: SenseVoice-Small for speech and SigLIP2 for images. Two-layer MLP projectors map those features into the MiniMind hidden space. On the output side, the Talker predicts several layers of Mimi codes at once using multi-token prediction, rather than one token at a time. Mimi itself is an 8-layer codebook running at 12.5 Hz over 24 kHz audio, and the Talker shares a common body with a light adapter on top of that codebook interface. Streaming generation, barge-in interruption and near-duplex interaction are handled with VAD. The practical consequence of the frozen encoders is that you inherit their behaviour. If SenseVoice-Small mishears an accent, no amount of MiniMind-O fine-tuning repairs it, because those weights are not being updated. That is a deliberate trade: it keeps the trainable surface small enough for one GPU.

Installing MiniMind-O and running a first inference

The README lists a single-command dependency install from requirements.txt, which pins transformers 4.57.6, gradio 5.49.1, modelscope 1.37.0 and funasr 1.3.1 among others. Torch is commented out in that file, so you install it yourself; the README points at the PyTorch stable wheel index and suggests checking torch.cuda.is_available() before going further. Clone the repository shallowly and install the pinned dependencies:

git clone --depth 1 https://github.com/jingyaogong/minimind-o
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

Inference needs more than the language weights. Four auxiliary models must sit in ./model: SenseVoice-Small, SigLIP2, Mimi and CAMPPlus. The README gives modelscope download commands for each and notes that you can instead git clone the corresponding repositories from the ModelScope or HuggingFace collections, which requires LFS. After this step the tree should show model/ with those four subdirectories and out/ holding llm_768.pth.

modelscope download --model gongjy/SenseVoiceSmall --local_dir ./model/SenseVoiceSmall
modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
modelscope download --model gongjy/mimi --local_dir ./model/mimi
modelscope download --model gongjy/campplus --local_dir ./model/campplus
modelscope download --model gongjy/minimind-3o-pytorch llm_768.pth --local_dir ./out

Once the released weights are in ./out, the CLI entry point is eval_omni.py with --load_from model and --weight sft_omni. The README also documents a transformers-format route: clone jingyaogong/minimind-3o from HuggingFace and pass the directory name to --load_from instead. For the Gradio demo there is a layout constraint worth reading twice. The web_demo_omni.py script scans ./scripts/ for subfolders containing weight files, so the transformers model directory has to be copied there first or the script errors out. That script is explicitly non-realtime: uploaded or recorded audio is resampled to 16k on the backend. Realtime conversation lives in webui/web_demo.py.

modelscope download --model gongjy/minimind-3o-pytorch --local_dir ./out
python eval_omni.py --load_from model --weight sft_omni
cp -r minimind-3o ./scripts/minimind-3o
cd scripts && python web_demo_omni.py

Training on the mini dataset in about two hours

Training runs from the trainer/ directory. The README's recommended mini pipeline starts with an SFT stage over sft_t2a_mini.parquet, using torchrun with a single process, learning rate 5e-4, batch size 40, max_seq_len 512, torch compile enabled, and --from_weight llm so the run starts from the base language checkpoint rather than from scratch. The mini dataset is the one to grab first: the README states that the mini split completes the full Thinker-Talker SFT path on a single RTX 3090 in roughly two hours, while the full split corresponds to the released weights. The training data covers T2A, I2T and A2A tasks, and the pipeline supports full-parameter training as well as audio projector, vision projector and DDP multi-GPU modes. The README's own hardware note lists eight RTX 3090s, so the two-hour figure is a single-card claim, not a description of the author's rig.

cd trainer && bash train.sh
CUDA_VISIBLE_DEVICES=0 torchrun --master_port 29560 --nproc_per_node 1 train_sft_omni.py --learning_rate 5e-4 --data_path ../dataset/sft_t2a_mini.parquet --epochs 1 --batch_size 40 --use_compile 1 --from_weight llm --save_weight sft_zero --max_seq_len 512 --use_wandb --use_moe 0

Where MiniMind-O is the wrong tool

The frozen encoders are the first hard boundary. SenseVoice-Small and SigLIP2 are not trained here, so the model's perception ceiling is their ceiling. The second is scale. A 0.1B backbone will not match a large hosted omni model on open-domain reasoning, and the README does not claim otherwise; it positions the project as the smallest complete omni implementation among public models, which is a statement about size, not quality. The third is the demo path. Because web_demo_omni.py resamples everything to 16k and is described as non-realtime, anyone evaluating latency by running that script will measure the wrong thing. Fourth, the repository has no releases retrieved, so there is no tagged version to pin against; the README's changelog shows a single 2026-05-05 entry for the initial open-source drop. If you need a supported product with an SLA, this is the wrong dependency. If you need to understand how an omni model is assembled, it is a reasonable one.

MiniMind-O versus a cascaded ASR plus LLM plus TTS stack

The alternative most teams already run is a cascade: Whisper-class ASR, then an LLM, then a TTS model, glued together with an orchestration layer. The difference is architectural, not cosmetic. A cascade has three independently swappable components, mature tooling for each, and a text transcript in the middle that you can log, redact and audit. MiniMind-O has one backbone, no intermediate transcript, and speech that stays in the hidden space from input to output. You gain prosody preservation and one fewer serialization step; you lose the ability to debug by reading what the model thought it heard, and you lose the option of replacing just the ASR stage. The Talker's multi-token prediction of Mimi codes is what makes streaming practical here, since predicting one code at a time would stretch the generation loop. Neither approach is strictly better. The cascade is easier to operate; MiniMind-O is easier to study as a single system.

Licence, maintenance and what an upgrade costs you

MiniMind-O is Apache-2.0, which permits commercial use and modification provided you keep the licence and attribution notices; this is a description of the licence text, not legal advice, and the bundled encoders and codecs may carry their own terms that you should check separately. On maintenance, the repository is not archived and the last push was on 2026-08-06, so it is recent. There are no retrieved releases, which means upgrades happen by pulling the master branch rather than by moving between tags. The practical cost of an upgrade is the checkpoint layout: the README's instructions assume specific directories under ./model and ./out, and the web demo assumes a model folder under ./scripts/. A change to any of those paths breaks the documented commands before it breaks the code. Dependencies are pinned exactly in requirements.txt, including transformers 4.57.6 and numpy 1.26.4, so a forced dependency bump is more likely to be your upgrade trigger than a repository change.

Editorial conclusion

Adopt MiniMind-O if you want to read, train and modify an end-to-end omni pipeline on a single 24GB GPU, and you accept that the released checkpoint is a 0.1B research artifact rather than a production voice assistant. Skip it if you need turnkey ASR accuracy or a supported API. Before committing, verify that the SenseVoice-Small, SigLIP2, Mimi and CAMPPlus checkpoints download into ./model, that torch.cuda.is_available() returns True, and that the mini parquet files land in ./dataset; the README does not document a rollback path for a failed training run.

Frequently asked questions

Official sources

1. Issues
2. jingyaogong/minimind-o on GitHub
3. License: Apache-2.0
4. Project website
5. README