Blaizzy/mlx-audio: TTS, STT and STS on Apple Silicon

What mlx-audio solves, and for whom

Speech models ship as research code. Each one has its own tokenizer, its own sampling loop, its own audio writing convention, and its own idea of what a voice argument looks like. If you want to compare Kokoro against Qwen3-TTS on the same sentence, you normally write two integration layers and maintain both.

mlx-audio collapses that into a single package built on Apple's MLX framework. The pyproject description states the intent plainly: inference of text-to-speech and speech-to-speech models locally on your Mac using MLX. The README lists text-to-speech, speech-to-text, speech-to-speech and music generation under one roof, with voice cloning, adjustable speech speed, an OpenAI-compatible REST API, and quantization at 3, 4, 6 and 8 bits.

The audience is narrow and specific. You need an M-series Mac. You need Python 3.10 or newer, since requires-python is ">=3.10". In exchange you get a uniform load_model and generate call across model families that otherwise share nothing, plus a CLI that turns a one-line shell command into a wav file. If you are building a Mac desktop app, a local dictation tool, or an offline audiobook pipeline, this is aimed at you. If your inference runs on a Linux box with an NVIDIA card, MLX does not run there and the project has nothing to offer you.

How the model loading and generation flow works

The architecture visible in the repository is a thin dispatch layer over per-model implementations. The Python API in the README imports load_model from mlx_audio.tts.utils and passes a Hugging Face repository id such as mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-8bit. That id is the whole configuration. The loader inspects the repo, picks the matching model class from mlx_audio, and returns an object exposing generate.

generate is a generator, not a function that returns a finished buffer. The README example iterates over it and each yielded result carries result.audio as an mx.array waveform, with result.audio.shape[0] giving the sample count. That design is what makes streaming possible: the CLI flag --stream emits audio as segments are produced, and --save additionally writes the streamed audio to disk. When generation produces multiple segments without --join_audio, the tool writes numbered files such as audio_000.wav and audio_001.wav.

Model weights are not vendored. Every supported model points at a Hugging Face repo, mostly under mlx-community, with some upstream repos such as bosonai/higgs-audio-v3-tts-4b. That means first run downloads weights, and huggingface_hub is a core dependency. Quantized variants are separate repos rather than a runtime flag, which is why the Kokoro row lists bf16, 8bit, 6bit and 4bit as distinct links. Choosing a quantization level is choosing a repo id.

Installing mlx-audio and generating a first wav

The README gives a plain pip install for the library. This pulls the core dependencies: mlx, huggingface_hub, miniaudio, numpy, scipy, sounddevice, tqdm and transformers. Note that transformers is pinned at ">=5.14.0", which is a recent major line, so check it against anything else in your environment.

pip install mlx-audio

To get the command line tool without touching your project environment, the README offers uv. The --prerelease=allow flag is required, which tells you the published versions are treated as pre-releases by the tooling.

uv tool install --force mlx-audio --prerelease=allow

With the CLI on PATH, the README's basic example generates speech from a text string using a named voice. The model id here is the Qwen3-TTS custom voice checkpoint. Expect a wav file in the working directory, and expect the first invocation to spend time downloading weights.

mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-8bit --text 'Hello, world!' --voice Vivian

Two flags are worth adding early. --play sends the result to the speakers through sounddevice, and --output_path writes into a directory you choose instead of the current one.

mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-8bit --text 'Hello!' --voice Vivian --play --output_path ./my_audio

For the web interface and REST API you need the server extra, and the README installs it from a clone rather than from PyPI. The extras are named dev, server, stt, tts, sts and all in pyproject.toml, so you can install only what a given deployment needs.

git clone https://github.com/Blaizzy/mlx-audio.git
cd mlx-audio
pip install -e ".[dev, server]"

The Python path mirrors the CLI. load_model returns an object whose generate method yields results; here the loop prints the sample count for each yielded chunk. The voice and lang_code arguments match the CLI flags.

from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-8bit")

for result in model.generate(
    "Hello from MLX-Audio!",
    voice="Vivian",
    lang_code="English",
):
    print(f"Generated {result.audio.shape[0]} samples")

The optional extras are where the real dependency weight sits

A bare pip install mlx-audio gets you the core, but not every feature. pyproject.toml splits the rest into extras, and the split is informative. The stt extra adds sentencepiece and zstandard. The tts extra adds mistral-common[audio] and sentencepiece. The server extra adds fastapi, uvicorn[standard], python-multipart and webrtcvad. The sts extra adds sentencepiece, mlx-lm and webrtcvad.

Two details stand out. First, webrtcvad appears in both server and sts, and both pin setuptools<81 with the comment that version 81 and later drop pkg_resources, which breaks webrtcvad. That is a real constraint you inherit: if another package in your environment demands a newer setuptools, you have a conflict to resolve. Second, the sts extra pulls mlx-lm because the speech-to-speech pipeline's default responder loads an arbitrary chat model. The comment in pyproject.toml notes that every other use of the language model code is vendored in mlx_audio/lm, so mlx-lm is only needed for that one path. Installing the all extra brings in everything at once, including zstandard, which the core install does not include.

Where mlx-audio is the wrong tool

The hard boundary is hardware. MLX is Apple's array framework for Apple Silicon, and the package depends on mlx>=0.31.1. There is no CPU fallback and no CUDA backend described anywhere in the project's documentation. A Linux CI runner cannot run this, which means you cannot easily test a mlx-audio pipeline in the same place you test the rest of your Python code. That is a genuine operational cost, not a footnote.

The second limitation is API stability. The project is at 0.5.4, with 0.5.3 and 0.5.2 both released within the same week in September 2026. The README documents the CLI surface in detail, but a library releasing at that cadence in the 0.x range is telling you the interfaces can move. If you are pinning mlx-audio inside a product with a long support window, budget for upgrade work.

The third is coverage. The README's model table is long, but the language column is uneven. KittenTTS is English only. Dia, OuteTTS and Soprano are English only. CSM and MisoTTS are English only. If your product needs reliable coverage across, say, twenty languages, you are choosing between Kokoro's eight, Chatterbox's twenty-three, and the larger Qwen3-TTS and OmniVoice checkpoints, and those are different models with different voice behaviour. The library gives you the choice; it does not make the choice equivalent.

How mlx-audio differs from running the same models in PyTorch

The obvious alternative is the original upstream implementations of each model, most of which are PyTorch and run on CUDA or CPU. The difference is not a feature list, it is where the computation happens. MLX targets unified memory on Apple Silicon, so the model and the activations live in the same memory the CPU sees, and the package ships quantized checkpoints at 3, 4, 6 and 8 bits as separate Hugging Face repos. A PyTorch implementation of the same model on a Mac either runs on CPU or through MPS, and you would manage quantization yourself with a different toolchain.

The practical consequence is portability in the opposite direction. A PyTorch pipeline you develop on a Mac can be deployed to a CUDA server with the same code. An mlx-audio pipeline cannot. You are trading deployment flexibility for a simpler local stack and access to the mlx-community quantized weights. There is also a Swift package mentioned in the README for iOS and macOS integration, which a PyTorch path does not give you at all. If your target is a shipped Mac or iOS app, that is a meaningful difference; if your target is a container on a rented GPU, it is irrelevant.

Licence and the cost of keeping up

The package itself is MIT, declared both in the README badge and in pyproject.toml as license = "MIT". That covers the code in this repository. It does not automatically cover the model weights, which are hosted on Hugging Face under mlx-community and, for some entries, upstream organisations such as bosonai. Each of those repos carries its own licence, and the README's model table links to them without stating terms. If you plan to ship generated audio commercially, check the licence on the specific checkpoint you load, not just on the library.

The upgrade cost is the release cadence. Three releases in the eight days before the last push on 2026-09-15, and a version number still below 1.0. The pyproject pins are loose in places (transformers>=5.14.0, mlx>=0.31.1, huggingface_hub>=1.0), so a fresh install can pull newer transitive versions than the author tested. The one pin that is deliberately tight, setuptools<81, exists because webrtcvad breaks otherwise. Reproducible builds here mean committing your lockfile, not trusting the ranges.

Editorial conclusion

Adopt mlx-audio if your inference host is an Apple Silicon Mac and you want one interface across Kokoro, Qwen3-TTS, Chatterbox, CSM and the rest. Skip it if you deploy on CUDA or CPU Linux, or need a stable API surface: the version is still 0.x and releases land weekly. Verify first that your chosen model repo resolves on Hugging Face, that the optional extra you need (stt, tts, server, sts, or all) is installed, and that your Python is 3.10 or newer.

Frequently asked questions

Official sources

1. Blaizzy/mlx-audio on GitHub
2. License: MIT
3. Project website
4. README
5. Releases