T5Gemma-TTS: an encoder-decoder LLM that speaks English, Chinese and Japanese

What T5Gemma-TTS actually solves

Most open TTS stacks ask you to choose between two things: a small, fast model that reads text in a fixed voice, or a large model you cannot retrain. T5Gemma-TTS takes the second path and then opens the training loop. The repository ships preprocessing, training, LoRA fine-tuning and inference scripts for a multilingual model that supports English, Chinese and Japanese, and it treats the voice as an input rather than a property of the checkpoint.

The intended user is someone who already has audio and wants a specific voice attached to new text. The README describes zero-shot cloning from reference audio, which means you pass a reference clip and its transcript at inference time instead of training a speaker embedding first. The second intended user is someone who wants to fine-tune: the repository lists full training, fine-tuning and LoRA fine-tuning, and the examples/ directory contains data_preprocess and training subfolders, so the pipeline from raw audio to a checkpoint is inside the repo rather than assumed.

Duration control is the feature that separates it from the usual cloning demo. You can state how long the output should be, which matters when you are dubbing to a fixed slot. If you leave the duration out, the README says the system estimates it from phoneme count and language-specific pacing rules, and it is explicit that the estimate is approximate.

The encoder-decoder mechanism and where the audio tokenizer sits

The architecture is a T5Gemma encoder-decoder language model repurposed for speech. The README's feature list and the arXiv link (2604.01760) are the source for that framing; the repository itself is Python and PyTorch, with models/, steps/ and hf_export/ directories holding the export path to HuggingFace format.

The practical consequence of the encoder-decoder split shows up in batch generation. The README states that generating multiple variations from the same input is efficient because the encoder runs only once and only the decoder runs in batch. That is a real architectural constraint, not a marketing line: the text side is computed a single time, and the sampling diversity comes from repeated decoder passes.

Audio does not come straight out of the language model. A separate tokenizer backend handles waveform encoding and decoding, and the requirements.txt pins a specific build: an xcodec2 tarball hosted at NandemoGHS/Anime-XCodec2-44.1kHz-v2. The README says this variant is the default because it better supports Japanese voices, and that for English and Chinese it recommends the original XCodec2 model instead. That switch is not cosmetic. The README notes you must install the original xcodec2 library with a versioned pip command when you use the original model, and the Gradio invocation then takes different model name and sample rate flags. Whisper appears as an auto-transcribe path, and the low-VRAM flags let you push both the codec and Whisper onto the CPU.

Installing T5Gemma-TTS and generating your first clip

The README gives a plain clone-and-pip route. Note that it instructs you to install a CUDA build of PyTorch before running the requirements file if you want GPU support, and requirements.txt itself caps torch and torchaudio at 2.8.0.

git clone https://github.com/Aratako/T5Gemma-TTS.git
cd T5Gemma-TTS
pip install -r requirements.txt

The model weights are not in the repository. They are pulled from HuggingFace at run time by name, so the first inference call downloads them. The simplest end-to-end check is the HuggingFace-format command-line script with a model directory and a target string:

python inference_commandline_hf.py \
    --model_dir Aratako/T5Gemma-TTS-2b-2b \
    --target_text "Hello, this is a test of the text to speech system."

You should get an audio file; the README's longer example adds --output_dir ./generated_tts to choose where it lands. To clone a voice, add a reference transcript and a reference waveform:

python inference_commandline_hf.py \
    --model_dir Aratako/T5Gemma-TTS-2b-2b \
    --target_text "Hello, this is a test of the text to speech system." \
    --reference_text "This is a reference." \
    --reference_speech path/to/reference.wav

If you would rather click than type, the Gradio UI takes the same model directory and a port:

python inference_gradio.py \
    --model_dir Aratako/T5Gemma-TTS-2b-2b \
    --port 7860

The repository also carries a Dockerfile and a docker-compose.yml. The compose file maps ${PORT:-7860} to 7860, reserves all NVIDIA devices, persists the HuggingFace cache in a named volume, and forwards a MODEL_DIR variable into the Gradio command. The Dockerfile takes a CUDA_VERSION build argument with cu118, cu121, cu124 and cu128 as options, defaulting to cu128. There is no install path in the README for anything other than pip and Docker, so treat those as the supported routes.

VRAM, quantization and the parts the README warns about

The memory floor is documented as a table. Full bfloat16 inference is listed at roughly 10.6 GB, an 8-bit encoder variant at roughly 8.6 GB, and a 4-bit encoder variant at roughly 7.6 GB. The README is careful about what is quantized: only the encoder, with the decoder left in full precision to preserve audio quality. If your assumption was that a 4-bit model means a 4-bit model end to end, that assumption is wrong here.

There are two other memory levers. The --cpu_codec flag moves the XCodec2 tokenizer to the CPU and the README claims roughly 3.5 GB saved, and --cpu_whisper moves Whisper to the CPU for roughly 5 GB. The --low_vram preset enables both and disables torch.compile. The README states these switches do not change model quality and only trade memory for latency on first runs. That is a plausible design, but it also means the low-VRAM path is slower in a way the README does not quantify.

The clearest limitation is platform support. The README has a Known Issues section stating that on some native Windows environments inference shows unstable behaviour, including inconsistent generation times and occasional hangs, that the root cause is still under investigation, and that WSL2 or Docker is the suggested workaround. The tested-environment table marks Linux with CUDA and Windows with CUDA under Docker as tested, plus Apple Silicon MPS on an M4 Max MacBook Pro. Native Windows is the gap. If your team develops on Windows laptops, that is a real constraint, not a footnote.

A second limitation is language coverage. English, Chinese and Japanese are the supported set. Nothing in the README suggests other languages, and the duration estimator's dependencies (langdetect, g2p_en, pyopenjtalk-plus, pypinyin) mirror those three.

How it differs from Kokoro TTS and other open TTS options

The useful comparison is with a small, fixed-voice TTS model rather than with a large cloning system. Kokoro TTS is the reference point many people arrive with: it is a compact model with a set of built-in voices, and the appeal is that it runs in modest memory and produces speech immediately. T5Gemma-TTS makes the opposite trade. It is a 2b-2b encoder-decoder language model with a separate audio codec, which is why the README quotes 7.6 to 10.6 GB of VRAM depending on quantization, and why the repository contains a training pipeline rather than only an inference script.

The difference in approach is where the voice lives. In a fixed-voice model, the voice is baked into the weights and selecting a speaker is a lookup. In T5Gemma-TTS, the voice is supplied at inference as reference audio plus a reference transcript, which is what makes zero-shot cloning possible and what makes the first call more expensive. The same split explains duration control: a fixed-voice model decides pacing internally, while T5Gemma-TTS lets you pass --target_duration and falls back to phoneme-count estimation when you do not.

The other comparison worth naming is the audio tokenizer, because it is a separate dependency with its own licence. The default Anime-XCodec2-44.1kHz-v2 build is tuned for Japanese, and the README recommends the original HKUSTAudio/xcodec2 model for English and Chinese. Swapping them changes a pip install, a model name flag and a sample rate flag. Anyone evaluating the project for English or Chinese should test both rather than assume the default is the best fit.

Licence, maintenance and what an upgrade costs

The repository code is MIT, and the README carries the badge to match. That covers the training and inference scripts. It does not automatically cover the model weights on HuggingFace or the XCodec2 tokenizer, which are separate artifacts with their own terms; the README links to the model card rather than restating a licence for the weights. If you plan to ship generated audio commercially, the tokenizer choice is the part to read up on, since the default and the alternative come from different publishers. This is not legal advice, and the repository does not resolve the question for you.

On maintenance, the last push to the default branch was on 2026-04-03. That is the same date as the most recent entry in the README's Updates section, which announced the technical report on arXiv. The earlier entry, dated 2025/12/17, added batch inference in the Gradio UI, 8-bit and 4-bit encoder quantization, the --low_vram option, Apple Silicon MPS support and PyTorch 2.9+ compatibility. No releases were retrieved for the repository, so versioning appears to happen through commits and through the HuggingFace model names rather than through tagged releases.

The upgrade cost is mostly in the dependency graph. requirements.txt pins transformers at 4.57.3 and omegaconf at 2.3.0, caps torch and torchaudio at 2.8.0, and pulls xcodec2 from a tarball URL rather than PyPI. A tarball dependency is harder to mirror in an air-gapped build than a normal package. The Dockerfile reduces that pain by installing everything at image build time and caching the HuggingFace models in a named volume, which is the route to take if you want reproducible environments rather than per-machine pip installs.

Editorial conclusion

Adopt T5Gemma-TTS if you need English, Chinese or Japanese speech with zero-shot cloning from a reference clip and you can give the model roughly 7.6 to 10.6 GB of VRAM, or run the --low_vram preset and accept slower first runs. Do not adopt it if you need a language outside those three, or if you are on native Windows, where the README documents unstable generation times and occasional hangs and points you to WSL2 or Docker instead. Before you build anything on it, verify the licence terms of the XCodec2 audio tokenizer you pick, since the default Anime-XCodec2 variant and the original HKUSTAudio/xcodec2 model come from different sources and only the repository's own code is MIT.

Frequently asked questions

Official sources

1. Aratako/T5Gemma-TTS on GitHub
2. Issues
3. License: MIT
4. Project website
5. README