Confucius4-TTS: Zero-Shot Voice Cloning Across 14 Languages Without a Reference Transcript

The problem Confucius4-TTS targets: one voice, many languages

Most voice cloning pipelines assume the reference clip and the target text share a language. Feed a Mandarin reference into an English synthesis and you typically get either a foreign accent or a voice that has drifted from the original speaker. Confucius4-TTS is aimed squarely at that gap. The README states the system preserves speaker identity across languages and produces unaccented speech, which is the harder half of the claim. The project lists 14 languages: Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay and Vietnamese. The README also notes more are coming, without naming them. The intended audience is fairly narrow. If you are building dubbing, audiobook narration, or a multilingual assistant where a single branded voice must speak several languages, this is the shape of tool you want. If you only ever synthesize one language, the cross-lingual machinery buys you nothing over a simpler single-language model. The second design decision worth noting is that no reference transcript is required. The README calls this unconstrained voice cloning. That matters operationally, because it removes a transcription step and a source of alignment errors from your pipeline. It also means the model has to infer content and speaker characteristics from audio alone, which is a harder task than transcript-conditioned cloning.

Speech encoder plus LLM: the two-stage pipeline visible in the repository

The architecture is described as a speech encoder plus large language model. The inference path splits into two stages, and the repository names them. The Text2Semantic stage, abbreviated T2S, is autoregressive and is where the LLM does its work. The README refers to the T2S stage running with HuggingFace Transformers by default, and to an alternative where vLLM accelerates that same stage using PagedAttention. So the flow is: text plus a reference WAV go in, the speech encoder turns the reference audio into conditioning the LLM can consume, the LLM generates semantic tokens autoregressively, and an acoustic stage turns those tokens into a waveform. The exposed Python API reflects this. ConfuciusTTS is constructed with a config path and a device, then generate() is called with text, lang, prompt_wav and a verbose flag, returning a tensor you save with torchaudio. The sample rate is exposed as model.sample_rate, which tells you the output rate is a property of the loaded model rather than a fixed constant you can hardcode. What the README does not specify is the acoustic decoder, the tokenizer vocabulary, the parameter count, or the encoder's identity. Those are in the linked arXiv paper, not in the repository text, so treat any claim about model size or token rates as unverified until you read the paper.

Installing Confucius4-TTS: Python 3.10, CUDA 12.6, three commands

The requirements are stated plainly: Python 3.10 and CUDA 12.6. Setup is a clone, a conda environment, and a pip install. The README gives these exact steps: git clone https://github.com/netease-youdao/Confucius4-TTS.git, then cd Confucius4-TTS, then conda create -n confuciustts python=3.10 -y, then conda activate confuciustts, then pip install -r requirements.txt. If your network cannot reach HuggingFace, the README instructs you to export HF_ENDPOINT=https://hf-mirror.com before running anything. That single environment variable is the difference between a first run that downloads weights and one that hangs. Basic synthesis goes through example.py with four flags: --prompt_wav, --text, --lang and --out, plus an optional --config pointing at config/inference_config.yaml. The Python API is short enough to quote in full: construct ConfuciusTTS with config_path and device, call model.generate(text=..., lang=..., prompt_wav=..., verbose=True), then torchaudio.save the result at model.sample_rate. A Gradio interface is included and starts with python webui.py --port 7860. The README notes the reference audio travels to the server over HTTP, so the browser and the server do not need a shared filesystem. That is a small detail with real consequences for anyone deploying this behind a load balancer.

The vLLM path and why it needs its own conda environment

The default T2S stage runs on HuggingFace Transformers. For faster generation the repository ships example_vllm.py, which swaps in vLLM and its PagedAttention kernel. The version constraint is explicit and tight: vLLM 0.16.0 with the V1 engine. The README states that older versions have not been tested and may not work, and gives the reason. The model registration and GPUModelRunner monkey-patches target v1 engine internals. That is a maintenance signal, not a bug. Code that patches a library's internal runner breaks when that library reorganizes its internals, and vLLM does that regularly. The installation advice follows from the same risk. The README recommends cloning a separate environment from the base one rather than installing vLLM into it, and warns that installing vLLM into the base env can break other packages. The commands are: conda create -n confuciustts_vllm --clone confuciustts, conda activate confuciustts_vllm, then pip install -r requirements_vllm_add.txt. The README also warns that vLLM has specific requirements on GPU architecture, driver and CUDA, and may not run on your hardware. Streaming is supported through a --stream flag on example_vllm.py, which calls model.generate_stream and, in the example, concatenates chunks into one WAV. If you need to start playing audio before synthesis finishes, that flag is the entry point. Note that the streaming example still writes a single file, so the chunked output is not exposed as a stream in the sample script.

Where Confucius4-TTS is the wrong tool

The cross-lingual design removes accents by intent. If your product needs an English speaker to sound recognizably English when reading English text, this engine is pushing in the opposite direction. The README frames unaccented cross-lingual synthesis as the feature. For dubbing a documentary where the narrator should keep a trace of origin, or for language-learning material where accent is the point, that behaviour is a defect. The vLLM backend is the second constraint. It is pinned to 0.16.0 and depends on monkey-patching GPUModelRunner, so a vLLM upgrade is not a routine dependency bump. Budget for reading the patch when you move versions. Third, the repository lists no releases. The README links a HuggingFace model page and a ModelScope page, and the install path pulls weights at runtime, but there is no versioned artifact on the repository side that tells you what changed between two points in time. If you need reproducible builds with pinned model revisions, you will have to pin the HuggingFace revision yourself, and the README does not show how. Fourth, the licence situation is ambiguous. The README badge says code_license Apache 2.0 and links to ./LICENSE, but the repository metadata reports NOASSERTION, meaning the licence could not be identified automatically. Those two signals conflict. Read the actual LICENSE file before you ship anything, and note that a code licence does not automatically cover downloaded model weights. The README does not state a separate model licence.

How this differs from a single-language cloning model such as XTTS-v2

Coqui XTTS-v2 is the closest well-known comparison point and takes a different approach. XTTS-v2 is built on a GPT-style autoregressive model conditioned on speaker embeddings, and its cloning is largely transcript-free in the same way. The meaningful difference is language coverage and how cross-lingual behaviour is framed. XTTS-v2 supports a fixed set of languages and its cross-lingual output often retains some accent from the reference speaker. Confucius4-TTS lists 14 languages and explicitly markets unaccented output as the goal, which suggests the training objective or data mix was chosen to suppress accent transfer rather than merely tolerate it. The second difference is the serving path. XTTS-v2 ships with its own inference server and its own runtime. Confucius4-TTS instead delegates the LLM stage to HuggingFace Transformers by default and offers vLLM as an accelerator, which means you inherit vLLM's batching and scheduling if you take that path, along with its version pinning. If you already run vLLM for other models, that is a real integration advantage. If you do not, it is an extra environment to maintain. Neither project publishes a head-to-head comparison in the material available here, so treat any quality difference as something you must measure on your own reference clips and target languages.

Maintenance cost and what the licence ambiguity means in practice

Three ongoing costs are visible from the repository alone. The first is the vLLM pin. Because the integration patches GPUModelRunner internals, every vLLM release is a potential break, and the README only claims support for 0.16.0 V1. The second is the absence of tagged releases. There is no changelog to diff, so tracking upstream changes means reading commits. The third is the model download itself. Weights come from HuggingFace or ModelScope at first run, and the README's mirror variable HF_ENDPOINT exists precisely because that download is a deployment concern in some networks. On licensing, the conflict between the Apache 2.0 badge and the NOASSERTION metadata is the thing to resolve first. Apache 2.0, if that is what ./LICENSE contains, permits commercial use and modification and requires you to retain notices. But the README does not state a licence for the model weights, and weight licences are frequently more restrictive than the code licence. If you plan to ship synthesized audio commercially, or to fine-tune on your own data, confirm the weight terms separately. This is not legal advice, and the repository text does not settle the question.

Editorial conclusion

Adopt Confucius4-TTS if you need one speaker identity carried across several of the 14 listed languages and you can supply a clean reference WAV without a matching transcript. Do not adopt it if you need a stable API surface, because the vLLM path depends on monkey-patches against vLLM 0.16.0 V1 engine internals, or if you need per-language accents preserved rather than removed. Before committing, verify three things: that your GPU, driver and CUDA combination satisfies vLLM's requirements, that the licence file at ./LICENSE matches the Apache 2.0 badge the README displays, and that your target languages are among the 14 listed rather than on the unspecified coming soon list.

Official sources

1. Issues
2. netease-youdao/Confucius4-TTS on GitHub
3. README