CosyVoice: the FunAudioLLM TTS stack, from 0.5B zero-shot cloning to a 150ms streaming path

What CosyVoice is for, and who actually needs it

CosyVoice targets a specific gap: teams that want a multilingual TTS model they can host themselves, clone voices from a short reference clip, and fine-tune on their own audio. The README describes Fun-CosyVoice 3.0 as "designed for zero-shot multilingual speech synthesis in the wild", and the language list backs that up: nine languages (Chinese, English, Japanese, Korean, German, Spanish, French, Italian, Russian) plus more than 18 Chinese dialects and accents, including Guangdong, Minnan, Sichuan, Dongbei and Shanghai. Cross-lingual cloning is supported, so a reference clip in one language can drive synthesis in another.

The intended user is not someone who wants a hosted API. It is an engineer with a GPU node who needs control over the voice, the text normalization pipeline and the deployment shape. The repository ships inference, training and deployment in one tree, with example directories for GRPO, LibriTTS and MagicData-RAMC. That breadth is the selling point and also the cost: you inherit a research repository's dependency surface, not a packaged library's.

If your requirement is a single English voice at low volume, CosyVoice is heavier than you need. The interesting capability here is multilingual coverage and cloning from short reference audio, not general-purpose synthesis.

How the CosyVoice pipeline is put together

The architecture follows the LLM-based TTS pattern. A language model predicts speech tokens, and a separate decoder turns those tokens into waveform. CosyVoice 2.0 introduced a 25Hz token rate, which is visible in the roadmap entries for the 25hz CosyVoice2-0.5B and 25hz CosyVoice-300M releases, and flow matching training support landed in July 2024. Fun-CosyVoice 3.0 sits on top of that lineage.

Two mechanisms are worth understanding before you evaluate it. The first is Repetition Aware Sampling, added in August 2024 for "llm stability". Token-level repetition is a known failure mode in autoregressive speech models, and RAS is the project's answer to it. The second is bi-streaming: the README states that CosyVoice supports both text-in streaming and audio-out streaming and reaches latency as low as 150ms. Streaming inference mode with kv cache and sdpa for RTF optimization is listed separately in the roadmap, so the streaming path has been iterated on across releases rather than bolted on once.

Text normalization is handled without a traditional frontend module, per the highlights section. That is a real architectural choice: instead of a rule-based text processor, the model is expected to read numbers, special symbols and varied text formats directly. Pronunciation inpainting for Chinese Pinyin and English CMU phonemes gives you a manual override when the model gets a reading wrong, which is the escape hatch that makes the no-frontend approach workable in production.

Installing CosyVoice and running a first synthesis

The README gives a Conda-based install. Clone with submodules, because the repository has a .gitmodules entry and the README warns that submodule fetches can fail on flaky networks.

git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
cd CosyVoice
git submodule update --init --recursive

Then create the environment. The README pins Python 3.10 and installs from requirements.txt through an Aliyun mirror.

conda create -n cosyvoice -y python=3.10
conda activate cosyvoice
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

The README notes that sox compatibility issues may require a system package. On Ubuntu the documented command is `sudo apt-get install sox libsox-dev`; on CentOS it is `sudo yum install sox sox-devel`.

Model weights come from ModelScope or Hugging Face. The README recommends downloading the pretrained checkpoints and the CosyVoice-ttsfrd resource. The ModelScope form is:

from modelscope import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('iic/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('iic/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')

For users outside China, the README gives the Hugging Face equivalent, using `huggingface_hub.snapshot_download` with the same local_dir values and the FunAudioLLM namespace for the Fun-CosyVoice3 and CosyVoice2 weights. After that, the repository root contains example.py, vllm_example.py and webui.py. The README does not print a full run command for example.py in the section shown, so the practical first step is to open example.py, confirm the model path matches the local_dir you downloaded to, and run it inside the activated environment. The webui.py entry point gives you a Gradio interface if you would rather click than script.

Where CosyVoice gets awkward: pins, platform skew and rollback

The dependency list is the first real constraint. requirements.txt pins torch==2.3.1, torchaudio==2.3.1, transformers==4.51.3, numpy==1.26.4 and pydantic==2.7.0, and it adds `--extra-index-url https://download.pytorch.org/whl/cu121`. That is a CUDA 12.1 build of PyTorch, so an existing environment on a different CUDA line will not simply absorb this file. TensorRT packages are pinned to 10.13.3.9 and are marked `sys_platform == 'linux'`, while onnxruntime switches between onnxruntime-gpu on Linux and plain onnxruntime on macOS and Windows. deepspeed is Linux-only as well. Windows and macOS users get a reduced feature set, not parity.

There is no CPU path documented in the README. Every runtime entry point named in the repository root assumes a working GPU stack, and the TensorRT and vLLM integrations are Linux-targeted. If your deployment target is a CPU-only container, this is the wrong project.

Rollback is the second gap. The README does not document model version pinning, checkpoint compatibility between Fun-CosyVoice3 and CosyVoice2, or how to revert a fine-tuned checkpoint to the base weights. Fine-tuning scripts exist under examples/, but the repository does not describe an upgrade path for a model you have already trained on. That matters because the roadmap shows a fast release cadence: 25hz CosyVoice2-0.5B in December 2024, vLLM support in May 2025, GRPO training and TensorRT-LLM runtime support in August 2025, and the Fun-CosyVoice3-0.5B-2512 base and RL models in December 2025. A moving target is fine for research; it is expensive for a production voice you have already tuned.

CosyVoice compared with other open TTS models

The README's own evaluation table is the most useful comparison, because it lists the competing open models side by side on the same test sets. On test-zh CER, Fun-CosyVoice3-0.5B-2512 scores 1.21 and the RL variant scores 0.81, against 1.45 for CosyVoice2, 1.52 for F5-TTS and 0.93 for VoxCPM. On test-en WER, Fun-CosyVoice3-0.5B-2512_RL scores 1.68, ahead of CosyVoice2's 2.57 and VibeVoice-1.5B's 3.04. Speaker similarity on test-zh is 78.0 for Fun-CosyVoice3-0.5B-2512, compared with 75.7 for CosyVoice2 and 74.1 for F5-TTS. These are the project's published numbers, not independent measurements.

The interesting comparison is model size against score. VibeVoice-1.5B, FireRedTTS2 at 1.5B, Index-TTS2 at 1.5B and HiggsAudio-v2 at 3B all carry more parameters than the 0.5B Fun-CosyVoice3 checkpoint, and on test-zh CER the 0.5B model lands at 1.21 while FireRedTTS2 sits at 1.14 and Index-TTS2 at 1.03. So the size advantage does not translate into a clean win on every metric. What the 0.5B checkpoint does give you is a lower memory floor and faster iteration, which is a different trade than raw accuracy.

Against Kokoro and Chatterbox, the README provides no scores, so no grounded comparison is possible. The one structural difference the README does establish is language coverage: nine languages plus 18+ Chinese dialects and accents, with cross-lingual cloning. If your workload is English-only, that coverage is dead weight and a smaller English-focused model will be easier to operate.

Licence, maintenance and what the repository does not promise

The code is Apache-2.0, and the LICENSE file sits at the repository root alongside CODE_OF_CONDUCT.md and FAQ.md. Apache-2.0 covers the source. It does not automatically settle the terms attached to the pretrained weights, which are distributed separately through ModelScope and Hugging Face under the FunAudioLLM and iic namespaces. The repository does not state model-weight licence terms, so a commercial deployment needs that checked at the model page rather than assumed from the code licence. That is a factual gap, not a legal opinion.

Maintenance activity is visible in the roadmap and the push history. The last push to the repository was on 2026-05-25, which is under six months before the current date, and the repository is not archived. The roadmap shows a steady stream of merged work through 2025, including third-party contributions such as the NVIDIA TensorRT-LLM runtime support credited to Yuekai Zhang. That pattern suggests the project is still receiving changes, though the README does not publish a support policy, a deprecation schedule or a compatibility matrix between model versions.

Upgrade cost is the practical question. Because requirements.txt pins exact versions of torch, transformers, onnxruntime and TensorRT, moving to a newer CosyVoice model means re-validating that whole stack. There is no documented migration guide between CosyVoice 1.0, 2.0 and Fun-CosyVoice 3.0 checkpoints. Budget for that re-validation as part of adoption, not as an afterthought.

Deployment surfaces: FastAPI, gRPC, vLLM and TensorRT-LLM

CosyVoice does not stop at a Python script. The repository root contains vllm_example.py and webui.py, and the tools/ and runtime/ directories hold the deployment code. The roadmap records a FastAPI server and client added in July 2024, and the requirements file pins fastapi==0.115.6, uvicorn==0.30.0, grpcio==1.57.0 and grpcio-tools==1.57.0, which indicates both HTTP and gRPC serving paths are part of the intended deployment story.

Accelerated inference has two documented routes. vLLM support for CosyVoice2-0.5B arrived in May 2025, and TensorRT-LLM runtime support with Triton arrived in August 2025. Those are the options if the default PyTorch path does not hit your throughput target. The README does not publish throughput numbers for either path, so the only latency figure you can rely on is the 150ms streaming claim, and that is stated for the bi-streaming mode rather than for batch synthesis.

A Gradio space is also listed in the roadmap for Fun-CosyVoice3-0.5B on ModelScope, which is useful for evaluating voice quality before you build anything. Treat it as an evaluation surface, not a production one.

Editorial conclusion

Adopt CosyVoice if you need a self-hosted multilingual TTS model with published training and inference scripts, and you are willing to pin a Conda environment around Python 3.10. Do not adopt it if you need a documented rollback path, a small CPU-only footprint, or a licence review you can finish without legal input: the repository does not document model-weight terms separately from the Apache-2.0 code licence. Before you commit, verify three things: that the ttsfrd resource or the WeTextProcessing fallback covers your text normalization needs, that your GPU node can satisfy the cu121 and TensorRT pins in requirements.txt, and that the 150ms latency figure holds for your own text lengths rather than the demo conditions.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. QwenAudio/CosyVoice on GitHub
5. README