Aratako/Irodori-TTS: Flow Matching TTS with Emoji Style Control

The gap Irodori-TTS fills: style control without a second model

Most open text-to-speech stacks split the problem in two. One model reads the words, another model, or a set of hand-tuned parameters, decides how those words should sound. Irodori-TTS collapses that split. The v4-Small checkpoint takes three conditioning branches at once: the text to be read, reference speech, and caption text. The caption is plain language, and the README's VoiceDesign example uses a Japanese phrase describing a calm speaker at a close distance. Emoji inside the input text are the other lever. The README lists emoji annotations as a feature that can influence delivery and non-verbal vocal expressions in supported checkpoints, and the inference examples mix emoji into the spoken string rather than passing them as a separate parameter.

The audience is narrow on purpose. This is a training and inference codebase, not a service. It ships infer.py, train.py, two Gradio apps and a manifest preparation script. If you want a voice you can clone from a thirty-second clip and then restyle with a sentence of description, that is the target use. If you want a drop-in replacement for a commercial TTS endpoint, the README points you elsewhere.

How the v4-Small architecture actually moves data

The pipeline is easier to reason about if you follow the audio, not the text. Reference clips are encoded into patched latent sequences by a reference latent encoder. Those latents condition speaker identity, and the README states v4-Small accepts up to 120 seconds of combined reference audio, so several short clips can be concatenated rather than one long one. The text and the caption both go through a single fine-tuned ModernBERT backbone. Separate text and caption projectors then map those shared encoder states into their own conditioning spaces, which is why one encoder can serve two roles without the two signals colliding.

Generation happens in latent space, not waveform space. A Diffusion Transformer with joint-attention blocks, Low-Rank AdaLN, half-RoPE and SwiGLU MLPs predicts continuous DACVAE latents, and the checkpoint's codec turns those latents back into 48 kHz audio. The released v2, v3 and v4 checkpoints use the 32-dimensional Semantic-DACVAE-Japanese-32dim codec. A duration predictor estimates output length, which is why the README can advertise automatic duration prediction and why the simple inference command has no length flag. The design follows Echo-TTS, and the README says so directly rather than presenting the architecture as novel.

Installing Irodori-TTS and running a first clone

The project uses uv and pins the PyTorch backend through mutually exclusive extras. Clone the repository, then sync one backend. The README's Linux and Windows CUDA 12.8 path is the first block below; AMD ROCm on Linux or WSL uses --extra rocm, Intel XPU uses --extra xpu, and CPU or macOS uses --extra cpu.

git clone https://github.com/Aratako/Irodori-TTS.git
cd Irodori-TTS
uv sync --extra cu128

After the sync, every later command is prefixed with uv run --no-sync. The README is explicit about why: running plain uv run would re-sync the environment without the backend extra you just selected. That detail matters more than it looks, because a silent re-sync is the kind of failure that produces a confusing import error rather than a clear message.

The shortest real use is a zero-shot clone from a reference file. The command below pulls the v4.1-Small checkpoint from Hugging Face, reads Japanese text, and writes a wav. Expect a download on first run and a GPU-bound generation step after it.

uv run --no-sync python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-v4.1-Small \
  --text "こんにちは、私はAIです。これは音声合成のテストです。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

If you have trained or downloaded a local checkpoint instead, swap --hf-checkpoint for --checkpoint and point it at a .pt or .safetensors file. To skip the reference clip entirely and let the model choose a voice, add --no-ref. For style control without a reference, pass a caption string alongside --no-ref, as in the VoiceDesign example. Two Gradio interfaces are available if you would rather click than type: gradio_app.py on port 7860 for reference-audio cloning and Speaker Inversion, and gradio_app_voicedesign.py on port 7861 for caption and reference conditioning.

Where Irodori-TTS will disappoint you

The reference audio limit is the first hard wall. The README states that one or more reference clips can be concatenated up to the checkpoint's 120-second limit. That is generous for cloning, but it is still a ceiling, and the README does not say what happens when you exceed it. There is no documented error message, truncation behaviour or warning for oversized reference input, so you should test that boundary yourself rather than assume it degrades gracefully.

Checkpoint and code versioning is the second trap. The main branch tracks the v4 and v4.1 codebase, including MeanFlow and a forthcoming v4-Large model, while staying backward-compatible with the released v2 and v3 base and VoiceDesign checkpoints. Older codebase states live under the v3, v2 and v1 tags. The README warns that v1 checkpoints and preprocessing are not compatible with v2, v3 or v4. Anyone who finds a v1 checkpoint and a current checkout is going to spend an afternoon on shape errors.

The third limitation is scope. There are no retrieved releases for this repository, so there is no changelog to read between commits. The README does not document rollback, and it does not document the OpenAI-compatible server beyond a pointer to the separate Irodori-TTS-Server project. If your requirement is a documented HTTP API with versioned endpoints, this repository is the wrong layer. It is training and inference code. Treat it that way.

How it differs from a conventional TTS toolkit

The obvious comparison is with a conventional acoustic-model-plus-vocoder pipeline, where you pick a speaker embedding, pick a vocoder, and control prosody through pitch and duration knobs. The difference here is that prosody and emotion are not knobs. They are conditioning signals. The caption branch and the emoji annotations both feed the same Diffusion Transformer that predicts latents, so style is expressed as a description or a symbol rather than as a numeric parameter you tune.

That trade has a cost. Numeric controls are reproducible: the same value gives the same result. Caption text is not, and the README does not claim it is. There is no documented mapping from a caption phrase to an acoustic outcome, and no list of which emoji are supported by which checkpoint. You are steering a generative model with language, which means you iterate by listening. Teams that need deterministic, auditable prosody control for something like a call centre will find that unacceptable. Teams prototyping character voices, or working in Japanese where the released codec and checkpoints are aimed, will find the iteration loop is the point rather than the problem.

Training, LoRA fine-tuning and what they cost to keep running

The repository is built for more than inference. train.py supports distributed runs through uv run --no-sync torchrun, with gradient accumulation, bf16 mixed precision and W&B logging. prepare_manifest.py builds the data manifest, and configs/ holds the configuration files, so a fine-tuning run is a YAML edit plus a torchrun invocation rather than a new codebase. For people who do not want a full run, the README lists PEFT LoRA fine-tuning for released checkpoints and Speaker Inversion, which learns reusable speaker embedding tokens for a target voice while freezing the base model. Those two options are the cheap path to a custom voice.

The upkeep is real. The dependency set pins torch, torchaudio and torchcodec to narrow ranges and requires transformers 5.12.1 or newer below 6. The SilentCipher watermarking dependency is pulled from a specific git commit rather than a released version, which means your lockfile is tied to that commit until upstream tags something. SilentCipher is applied when available, per the README, so watermarking is a runtime condition rather than a guarantee.

On licensing: the repository code is MIT, and pyproject.toml declares license = "MIT". That covers the code you are reading. It does not automatically cover model weights, the DACVAE codec, or the SilentCipher dependency, which come from other sources with their own terms. Check each of those separately before you ship generated audio, and do not read the MIT badge on the code as a blanket clearance.

Editorial conclusion

Adopt Irodori-TTS if you can run PyTorch 2.10 or newer on a CUDA, ROCm or XPU machine, you want zero-shot cloning from your own reference audio, and you are willing to read the v4.1 model card on Hugging Face because the README does not cover every flag. Do not adopt it if you need a hosted API with a stable contract: the repository points at a separate Irodori-TTS-Server project for that, and the README does not describe its endpoints. If you are on macOS, sync the cpu extra first and check that MPS inference actually completes before you plan around it. Before committing, verify three things: that the checkpoint you want matches the code tag you checked out, since v1 checkpoints and preprocessing are not compatible with v2, v3 or v4; that your reference audio fits inside the 120-second combined limit; and that your intended use is compatible with the SilentCipher watermarking path, which the README says is applied when available.

Frequently asked questions

Official sources

1. Aratako/Irodori-TTS on GitHub
2. Issues
3. License: MIT
4. Project website
5. README