voxtral-mini-realtime-rs: Voxtral ASR and TTS in Rust, natively or in a tab

A Rust port of Mistral's speech models on Burn

voxtral-mini-realtime-rs reimplements two Mistral models in Rust using the Burn machine learning framework: the Voxtral Mini 4B Realtime model for streaming speech recognition, and the Voxtral 4B TTS model for speech synthesis. The README describes it as streaming speech recognition and text to speech running natively and in the browser.

The framework choice is what makes this unusual. Burn provides the tensor operations and the autodiff and autotune machinery, and cubecl at 0.9 carries the WGSL compute path. The model code itself follows the standard pattern: safetensors for the BF16 weights, serde and serde_json for configuration, and a Burn tensor matmul for the linear layers in the full precision path.

The audio stack is conventional Rust. hound handles WAV input and output, rubato handles resampling with the audioadapter-buffers crate providing the buffer adapters, and rustfft produces the spectrograms. Tokenisation uses tiktoken-rs, described in the feature list as Tekken BPE encoding that is WASM compatible and needs no C dependencies, which matters because the browser build cannot rely on a native extension.

The audience is developers who want local speech inference without provisioning a Python environment and a GPU machine, and who are willing to accept the model download as the cost of entry.

Two inference paths and the Q4 GGUF route

The README lays out the two paths side by side, and the choice between them is the central design decision of the project.

The BF16 path uses SafeTensors weights of roughly 9 GB, performs linear operations with the Burn tensor matmul, keeps embeddings as an f32 tensor of 1.5 GiB, and does not run in the browser. The Q4 GGUF path uses GGUF Q4_0 weights of about 2.5 GB for ASR and 2.67 GB for TTS, runs the linear operations through a custom WGSL shader that fuses dequantisation with the matmul, keeps embeddings quantised on the GPU at 216 MB with CPU bytes for lookups, and does run in the browser.

The shader work is described in the architecture notes: custom WGSL compute shaders with vectorised u32 reads and vec4 dot products, and a dual-path kernel dispatch where a shared memory tiled kernel handles single token decode while a naive kernel handles multi row encode and prefill. The listed optimisations for the TTS path are batched CFG at batch size 2, fused QKV plus gate and up projections, and a pre-allocated KV cache.

The point of the Q4 route is that a 4B parameter model becomes something a browser tab can hold, and the README states that the Q4 GGUF models run entirely client side through WASM and WebGPU with no server in the loop.

What the benchmarks actually say

The README publishes measurements on an NVIDIA DGX Spark with GB10 and LPDDR5x memory, and they are worth reading carefully because they differ sharply by path.

For recognition, using 16 seconds of test audio averaged over three runs, the Q4 GGUF native path takes 1021 ms to encode and 5578 ms to decode, 6629 ms in total, which is a real time factor of 0.416, 19.4 tokens per second and 703 MB of memory. The BF16 native path encodes faster at 887 ms but decodes far slower at 23689 ms, giving a total of 24607 ms and an RTF of 1.543, with 9.2 GB of memory. The Q4 GGUF path compiled to WASM takes about 225 seconds for the same audio, an RTF of roughly 14.1 and about 0.5 tokens per second, in the browser.

For synthesis, the quick brown fox sentence at nine tokens with the casual_female voice, Q4 GGUF native with three Euler steps takes 3.7 seconds to produce 3.84 seconds of audio, an RTF of 0.97 and a 2.67 GB model. The WASM variant with eight Euler steps takes 367 seconds for 3.52 seconds of audio, an RTF of 104.

Quality numbers are given too. The README reports 8.49 percent word error rate on FLEURS English over 647 utterances, against the 4.90 percent Mistral reports for f32, and notes that the Q4 model at three Euler steps reaches real time with a Whisper large-v3 transcription of the output. Model load time is given as 3.9 s native and 9.2 s in WASM, including the shard download over localhost.

The Q4 padding bug that upstream introduced

One section of the README is worth quoting in outline, because it is the kind of detail that decides whether a port works at all.

The upstream mistral-common library left-pads audio with 32 silence tokens at 12.5 Hz. After the mel, convolution and reshape stages, that padding covers only 16 of the 38 decoder prefix positions, and the remaining 22 contain actual audio. The f32 model copes with this, but Q4_0 quantisation makes the decoder sensitive to speech content in the prefix. Audio that starts immediately with speech, which is the normal case for a microphone recording or a clip with no leading silence, then produces all-pad tokens instead of text.

The fix is to increase the left padding to 76 tokens, which maps to exactly 38 decoder tokens of silence and covers the full streaming prefix. The details live in src/audio/pad.rs.

This is a good illustration of what a port like this actually involves. The hard problems are not the transformer, they are the places where a quantised model is less forgiving than the full precision one, and where an upstream library makes an assumption that only shows up after a month of silence has been removed.

Running the CLI and the browser build

The native path starts with fetching weights. The ASR models are pulled with the Hugging Face CLI:

uv run --with huggingface_hub \
  hf download mistralai/Voxtral-Mini-4B-Realtime-2602 --local-dir models/voxtral
uv run --with huggingface_hub \
  hf download TrevorJS/voxtral-mini-realtime-gguf --local-dir models/

Transcription then runs through the CLI binary with the wgpu, cli and hub features:

cargo run --release --features "wgpu,cli,hub" --bin voxtral -- \
  transcribe --audio audio.wav --gguf models/voxtral-q4.gguf

Synthesis uses the same binary with a speak subcommand, a voice name and an optional --gguf path, and --euler-steps tunes the speed and quality trade-off. There are 20 preset voices across 9 languages, and speak --list-voices prints them.

The browser build is a separate target. The Makefile has a build-wasm target, and the README shows the same command:

wasm-pack build --target web --no-default-features --features wasm

WebGPU requires a secure context, so the browser demo needs a self signed certificate, and the README gives the openssl invocation and then bun serve.mjs to start the dev server, after which you open https://localhost:8443, accept the certificate and click Load from Server to pull the model shards. Two hosted demos on Hugging Face Spaces cover the same ground without any of that setup.

Engineering habits visible in the repository

The tree tells you how seriously the project takes correctness, which matters more for a speech model than for most libraries.

The Makefile is organised as a set of named checks. build, build-release and build-wasm cover the three build targets. lint runs clippy with warnings denied for the native target and a separate lint-wasm target for wasm32-unknown-unknown. test runs the cargo test suite, and there are benchmark targets for the audio path and for the Q4 operations, plus an end to end benchmark binary. Profiling is built in with a chrome tracing profile and a flamegraph target, the trace being written to trace.json for chrome://tracing or Perfetto. There are evaluation targets for word error rate on FLEURS and on LibriSpeech.

Around that sit the directories that show the scope of the work: benches/, fuzz/, tests/, scripts/, patches/, docs/, web/, and two Hugging Face Space directories, space/ and space-tts/, which are the hosted demos. A playwright configuration exists for browser testing, with @playwright/test as the only JavaScript dependency in the package manifest.

The Cargo feature list also shows how the binary is kept small for each target. The default features are wgpu and native-tokenizer, the wasm feature set pulls in wasm-bindgen, wasm-bindgen-futures, js-sys, web-sys, console_error_panic_hook and the getrandom wasm_js backend, and an optional profiling feature adds tracing-chrome.

Where voxtral-mini-realtime-rs is the wrong tool

The limitations are numeric and the README is upfront about them.

The browser path is not interactive for recognition. An RTF of about 14.1 means roughly fourteen seconds of compute per second of audio, and the WASM synthesis path at RTF 104 is far worse. The browser build is a demonstration that a 4B model fits in a tab, and it is also a route for privacy sensitive transcription where waiting is acceptable. If you need a live caption or a live voice, the native path is the only one of the three in the table that is close to real time, and even that is at RTF 0.416 for the Q4 GGUF build.

Second, the quality gap is real. The Q4 path is reported at 8.49 percent word error rate on FLEURS English, nearly double the 4.90 percent that Mistral reports for the f32 model. The README also states that the upstream service has retired the expert and vision modes, so there is effectively one model and one quality tier to work with here.

Third, the weight download is not a small thing. Around 2.5 GB for ASR and 2.67 GB for TTS in Q4, and roughly 9 GB and 8 GB respectively for BF16, plus a native build of Rust with a GPU backend, or for the browser a WebGPU capable browser and a secure context. A team that wants a hosted transcription endpoint, or that needs consistent accuracy across a fleet, has a different tool for that.

One release-hygiene note: the tagged versions v0.2.3, v0.2.4 and v0.2.5 are all dated 2026-04-02, while the last push was on 2026-10-12, so the published crate and the repository head are not on the same clock.

Editorial conclusion

Adopt voxtral-mini-realtime-rs if you need streaming speech recognition or synthesis on hardware you control, want the transcription to stay on the machine, and are willing to download a multi gigabyte model and work with a Rust toolchain and WebGPU. Do not adopt it for latency sensitive browser use, since the README measures the WASM path at roughly 14.1 RTF for recognition and 104 for synthesis, and do not adopt it expecting parity with the full precision model, since the README reports 8.49 percent word error rate on FLEURS English against the 4.90 percent Mistral reports for f32. Verify three things first: that your GPU and driver support the wgpu backend, that you have disk and bandwidth for the 2.5 GB Q4 ASR weights or the 2.67 GB TTS ones, and that the browser path is acceptable for your use case, since it needs a secure context for WebGPU and a 2 GB allocation limit workaround. The licence is Apache-2.0 and the last push was on 2026-10-12.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. TrevorS/voxtral-mini-realtime-rs on GitHub