# Hibiki simultaneous French-to-English speech translation with local backends

> Hibiki generates timestamped text and target-language audio while French speech is still arriving. It offers PyTorch, MLX, MLX-Swift, and Rust inference paths, but remains limited to French-to-English translation.

**kyutai-labs/hibiki** — Hibiki is a model for streaming speech translation (also known as simultaneous translation). Unlike offline translation—where one waits for the end of the source utterance to start translating--- Hibiki adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. 

- Repository: https://github.com/kyutai-labs/hibiki
- Stars: 1,531 · Forks: 118
- Language: Rust
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/kyutai-labs-hibiki

## Hibiki's streaming translation boundary

Hibiki addresses the delay imposed by offline speech translation. An offline system can wait for the end of a source utterance, while Hibiki processes speech chunks and generates target-language text and audio while the speaker continues. It is therefore aimed at researchers studying simultaneous speech translation, developers prototyping conversational translation interfaces, and teams with a specific French-to-English requirement.

The project describes target speech generation as optionally preserving the source speaker's voice. It also produces timestamped text alongside a continuous audio stream. Those properties make it different from a conventional text machine-translation API, but they also narrow its current role. The README states that Hibiki supports only French-to-English translation. There is no documented support for reverse translation or other language pairs.

Hibiki-M is identified as a smaller alternative intended for local smartphone use. That statement points toward on-device use, but the documentation does not provide a separate Hibiki-M installation command, resource table, or quality comparison. The larger Hibiki 1B and 2B releases are the concrete model choices covered by the listed inference packages.

## How Hibiki's multistream decoder moves audio and text

Hibiki is a decoder-only model built around Moshi's multistream architecture. Source and target speech are modeled jointly, allowing the model to keep reading the source stream while producing translated speech. The README states that Hibiki emits text and audio tokens at a constant rate of 12.5 Hz. This fixed token cadence supports continuous output audio and lets the system attach timing information to translated text.

The latency strategy is contextual rather than absolute. Hibiki accumulates enough source context to decide when target output is predictable, then continues that process chunk by chunk. This matters because emitting every possible token immediately can produce premature translations, while waiting too long recreates the delay of offline translation. The project's stated goal is to find a usable point between those two extremes.

Inference uses temperature sampling instead of a more complicated policy. The README says this makes batching compatible with Hibiki. Voice transfer is adjusted through Classifier-Free Guidance and `--cfg-coef`. The default is 1, and the README describes 3 as a typical value. Increasing the coefficient can improve voice similarity, but excessive values can reduce translation quality. That trade-off should be treated as a quality control decision, not simply a fidelity dial.

## Synthetic contextual alignment behind the Hibiki models

Hibiki's training data is the part of the project most likely to shape its behavior in production. The model requires aligned French speech, English speech, and English text from the same speaker, but the README says such data is not available in significant quantities. Kyutai therefore uses synthetic data generation.

The process begins with an off-the-shelf MADLAD machine translation system. Word-level matching between source and target transcripts produces a contextual alignment. Its operating rule is that a target word should appear only after it becomes predictable from the source. The alignment is then applied in two documented forms. One form inserts silence. The other synthesizes target speech with voice-controlled, alignment-aware text-to-speech.

This design ties translation timing directly to speech generation. Inserted silence can postpone an output token without waiting for a naturally aligned recorded target utterance. The approach also explains why the system can expose timestamped translation rather than a transcript generated only after a complete input. It does not, however, establish how well the model handles speakers, accents, noise, or domain vocabulary absent from the synthetic training distribution. The README does not publish those evaluations.

## PyTorch and MLX setup for a first French audio translation

The shortest documented route starts from the `moshi` PyTorch package. Install it, retrieve the sample named by the README, and invoke the Moshi inference module with the Hibiki 1B bf16 repository. The output path is explicitly named `out_en.wav`.

```bash
pip install -U moshi
```

The installation adds the inference package used by the README. It does not download the model at installation time; the `--hf-repo` argument selects the model repository when inference begins.

```bash
wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
python -m moshi.run_inference sample_fr_hibiki_crepes.mp3 out_en.wav --hf-repo kyutai/hibiki-1b-pytorch-bf16
```

The first command downloads the sample MP3. The second reads that file and writes translated audio to `out_en.wav`. The README also describes timestamped text output, but this particular command does not print a separate text file, so it should not be presented as producing one.

For Apple-oriented local experimentation, the documented package is `moshi_mlx`, with version `0.2.1` or newer required. The repository identifier and input and output arguments differ from the PyTorch command.

```bash
pip install -U moshi_mlx
wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
python -m moshi_mlx.run_inference sample_fr_hibiki_crepes.mp3 out_en.wav --hf-repo kyutai/hibiki-1b-mlx-bf16
```

The README does not state Python version or memory requirements for these commands. The root `pyproject.toml` declares `requires-python = ">= 3.10"` for the `hibiki` package itself, but that package depends on `moshi >= 0.1.0` and should not be conflated with the standalone installation instructions.

## Rust, Metal, CUDA, and real-time MLX execution

The Rust implementation is a simple file-to-file application under `hibiki-rs`. It supports an NVIDIA CUDA build and an Apple Metal build. The documented command downloads the same French sample, changes into the Rust directory, and releases the application rather than using a development-profile run.

```bash
cd hibiki-rs
wget https://github.com/kyutai-labs/moshi/raw/refs/heads/main/data/sample_fr_hibiki_crepes.mp3
cargo run  --features metal -r -- gen sample_fr_hibiki_crepes.mp3 out_en.wav
```

The command's double space before `--features` is harmless, but the exact documented feature and generated subcommand are important. For an NVIDIA system, the README identifies the alternative as `--features cuda`; it does not show the complete command, operating system setup, or supported driver versions.

MLX also has a local web interface intended for real-time use. Unlike the file-based examples, it does not take sample or output paths in the documented command.

```bash
python -m moshi_mlx.local_web --hf-repo kyutai/hibiki-1b-mlx-bf16
```

This starts the local web UI and selects the 1B MLX bf16 model. The README does not document the listening device, microphone permissions, browser port, latency measurements, or production hardening. Mobile support comes through the separate `kyutai-labs/moshi-swift` repository, which the README describes as experimental and says was tested on an iPhone 16 Pro.

## Hibiki model sizes, licensing, and maintenance limits

The repository exposes Hibiki 1B and Hibiki 2B variants in PyTorch and MLX bf16 formats. The README describes 1B as the on-device option and lists 8 RVQ layers per stream. It lists 16 RVQ layers per stream for 2B. It does not provide exact memory consumption, startup time, real-time factor, or comparative quality results, so the model names alone are not enough to estimate deployment requirements.

Licensing varies by component. The repository identifies `Apache-2.0` as the project license and contains `LICENSE-APACHE` and `LICENSE-MIT`. Its root `pyproject.toml` declares `MIT` for the Python package. Separately, the README says all released model weights are under `CC-BY 4.0`. That distinction matters for an engineer: code packaging and model weights cannot be treated as one blanket grant without reviewing the applicable files and notices. This is a licensing inventory, not legal advice.

The last push was on 2026-09-09, so the repository had recent source activity within the six-month maintenance window. However, it has no GitHub releases. Installation examples depend on mutable package names and Hugging Face repositories rather than documented Hibiki release versions, which increases the importance of pinning whatever artifacts an application elects to use. The README also says quantized models are coming soon, so they are not an available deployment option in the documented list.

## Why an offline ASR translation TTS pipeline is different

A conventional alternative is a staged pipeline: automatic speech recognition, machine translation, and text-to-speech. Such a pipeline can offer broader language and provider choices, especially when an offline batch transcript is acceptable. Its translation is not generated as part of one joint speech model, and target audio begins only after the intermediate stages produce suitable output. That architecture does not provide the same continuous multistream interaction described for Hibiki.

Hibiki instead couples source speech, target speech, and target text in a decoder-only model. Its advantage is architectural alignment with simultaneous translation: source and target streams are processed together at 12.5 Hz, with contextual timing learned during training. Its limitation is equally specific. The current system is documented only for French-to-English, while a component pipeline may be preferable when multilingual routing, editing of recognized text, or independent replacement of translation and speech services matters.

For production engineering, the deciding evidence should be backend-specific output inspection rather than the word "streaming." Compare start-up delay, translation pauses, text timing, voice similarity, and translation errors on the same French recordings. The README gives a voice-guidance setting but does not report those measurements, so it cannot establish which backend or coefficient is best for a particular workload.

## Conclusion

Researchers evaluating simultaneous speech-to-speech translation or developers targeting supported Apple and NVIDIA environments should inspect Hibiki's latency, voice similarity, and output quality before adopting it. Organizations needing another language pair, broad platform support, or a production package release should reject it for now. Verify the intended backend, model license, memory requirements, and FR to EN limitation first.

## FAQ

### Which language directions does Hibiki currently support?

Hibiki currently supports only French-to-English translation. The repository lists Hibiki 1B and Hibiki 2B models for that direction across PyTorch and MLX backends.

### Can Hibiki run locally on an iPhone or Mac?

The README says the smaller Hibiki-M can run locally on smartphone hardware and points to the separate moshi-swift repository for iPhone execution. It also provides an MLX path for macOS, while describing the Swift implementation as experimental.

### What does the Hibiki cfg-coef setting change?

The `--cfg-coef` parameter controls voice-transfer fidelity. The default is 1 and the README calls 3 a typical value; excessively high coefficients can worsen translation quality.

## Sources

- [Issues](https://github.com/kyutai-labs/hibiki/issues)
- [kyutai-labs/hibiki on GitHub](https://github.com/kyutai-labs/hibiki)
- [License: Apache-2.0](https://github.com/kyutai-labs/hibiki/blob/main/LICENSE)
- [README](https://github.com/kyutai-labs/hibiki/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/kyutai-labs-hibiki