GLM-TTS: A Two-Stage LLM and Flow-Matching Speech System With Phoneme-Level Control

The Polyphone Problem and the 3-Second Cloning Problem

Two concrete problems sit behind this project. The first is pronunciation ambiguity. The README names the character 行, which can be read as xíng or háng, and states that the Phoneme-in mechanism exists to address automatic pronunciation ambiguity in polyphones and rare characters. That is a narrow, testable failure mode, and it is the kind of thing that breaks educational assessment and audiobook pipelines where a wrong reading is not a stylistic quirk but a defect. The second problem is speaker adaptation. GLM-TTS states that zero-shot voice cloning works from 3 to 10 seconds of prompt audio, with no fine-tuning for specific speakers. That matters for anyone who has a short recording and no budget for a training run. The intended audience is therefore fairly specific: teams building Chinese-language speech output, with English mixed text supported as a secondary case, who need either a particular voice or a particular pronunciation, and who are willing to run inference themselves. It is not aimed at people who want a hosted API, and it is not aimed at low-resource language coverage, since the README states Chinese is primary.

Two Stages: Llama Generates Tokens, Flow Matching Generates Audio

The architecture is a pipeline with a clean split. Stage one is a large language model based on Llama architecture that converts input text into speech token sequences. Stage two is a Flow Matching model that converts those token sequences into a high-quality mel-spectrogram, after which a vocoder produces the audio waveform. Speaker identity enters through prompt audio: the system extracts speaker features from that audio rather than adjusting model weights. Because the first stage is an autoregressive token generator, streaming inference is possible, and the README lists it as a supported feature for interactive applications. The phoneme path is the more interesting mechanism. Training applies random G2P conversion to parts of the text, which the README says compels the model to adapt to hybrid input sequences while preserving its ability to understand pure text. At inference the workflow is G2P, then table lookup replacement, then hybrid input. A dynamic controllable dictionary identifies polyphones or rare characters and swaps in specified target phonemes, and the replaced phonemes are fed alongside the original text. The result is targeted control rather than full phonetic transcription, which is the right trade-off if you only care about a handful of words per script.

Getting It Running: Commands, Weights and the NPU Fork in the Road

The README gives two install paths. On GPU: clone the repository, cd into it, and run pip install -r requirements.txt. On NPU, the instructions change shape entirely. You pull a CANN image, quay.io/ascend/cann:8.5.1-910b-ubuntu22.04-py3.11, and run it with a specific set of device mounts: /dev/davinci_manager, /dev/devmm_svm, /dev/hisi_hdc, plus bind mounts for /usr/local/dcmi, /usr/local/bin/npu-smi, the Ascend driver lib64 directory, version.info, and /etc/ascend_install.info. The DEVICE variable is set to a davinci device such as /dev/davinci7. Inside that container you set an extra index URL pointing at the PyTorch CPU wheel index and install requirements_npu.txt with --no-build-isolation. Python must be 3.10 through 3.12. Weights come from either HuggingFace or ModelScope into a ckpt directory: huggingface-cli download zai-org/GLM-TTS --local-dir ckpt, or modelscope download --model ZhipuAI/GLM-TTS --local_dir ckpt. Inference has three entry points: python glmtts_inference.py with --data=example_zh, --exp_name=_test and --use_cache, plus an optional --phoneme flag; a shell wrapper at glmtts_inference.sh; and a Gradio interface via python -m tools.gradio_app. The reinforcement learning dependencies are separate and optional: cd grpo/modules, then git clone s3prl and LaughterSegmentation, then place wavlm_large_finetune.pth in grpo/ckpt. Note that this step pulls code from two third-party GitHub repositories at install time.

The Emotion Claims Rest on Weights That Are Not Here Yet

The title of the project promises controllable and emotion-expressive synthesis via multi-reward reinforcement learning, and the feature list repeats that RL-enhanced emotion control produces more natural emotional expression and prosody control. The news section undercuts that. Two items are marked Coming Soon: a 2D Vocos vocoder update, and model weights optimized via reinforcement learning. So the released artifact is the base inference stack plus the GRPO module tree, not a finished RL-aligned checkpoint. The GRPO setup is also the heaviest part of the install and the least self-contained, since it depends on cloning s3prl and LaughterSegmentation and sourcing a WavLM checkpoint by hand. If your evaluation is about emotional expressiveness, you are evaluating a moving target. The honest reading is that the architecture and the phoneme mechanism are what you can assess today, and the RL story is documented in the arXiv report 2512.14291 rather than reproducible from the repository as shipped.

Where GLM-TTS Is the Wrong Choice

Language coverage is the first boundary. The README states Chinese is primarily supported, with English mixed text also supported. If you need monolingual English at production quality, or any language outside that pair, this is not the tool. The second boundary is operational. There is no retrieved release, so there is no tagged version to pin, no changelog to diff, and no upgrade contract. The last push date tells you the repository is active, but activity is not the same as a stable interface, and the inference script arguments could change without a version bump. The third boundary is hardware. The NPU path is documented in detail but requires a specific CANN image and a list of device mounts, which means the deployment surface is wider than a plain pip install. The fourth is the phoneme feature itself: it is opt-in behind a flag, and the README describes a dynamic controllable dictionary without publishing its contents, so you should expect to build and maintain that dictionary for your own domain vocabulary. The fifth is that streaming and quality are both claimed without published latency or MOS figures in the material available here, so treat them as design intent rather than measured properties.

How This Differs From a Single-Stage Vocoder Pipeline

Most open speech synthesis stacks you would compare against are end-to-end: text features go into an acoustic model and out comes a waveform, with one training objective and one inference graph. GLM-TTS splits that into an autoregressive language model over discrete speech tokens and a separate flow-matching stage that maps tokens to a mel-spectrogram before the vocoder. The practical difference is where you can intervene. In a single-stage system, correcting one word's pronunciation usually means retraining or post-editing audio. Here, the token stage accepts hybrid phoneme-plus-text input, so the intervention happens before synthesis, through a dictionary lookup, and the rest of the utterance is untouched. That is a real architectural distinction, not a marketing one. The cost is a longer chain: tokenizer, LLM, flow model, vocoder and frontend all have to be present and compatible, which is why the README tells you to download all five components and why the ckpt directory is the first thing to check when something fails. A single-stage pipeline has fewer places to break; this one has more places to adjust.

Licence, Maintenance and What an Upgrade Actually Costs

The repository is Apache-2.0, which permits commercial use and modification, and the model weights are hosted separately on HuggingFace and ModelScope under the zai-org and ZhipuAI namespaces. The licence identifier in the repository metadata does not automatically tell you the terms attached to the downloaded checkpoints, so check the model card on whichever host you pull from before shipping. This is not legal advice. On maintenance, the cost profile is unusual. The core install is a single requirements.txt, which is cheap. The RL path is not: it clones s3prl and LaughterSegmentation from GitHub at a point in time, and a git clone without a pinned commit is an unpinned dependency that can change under you. If you do not need the RL tooling, skip grpo/modules entirely and you avoid that whole surface. If you do need it, pin both repositories to specific commits in your own build. The Coming Soon items mean the project will likely ship new weights and a new vocoder, and because there are no releases, there is no version boundary to gate that change behind. Budget for re-validating your phoneme dictionary and your audio output whenever you pull from main.

Editorial conclusion

Adopt GLM-TTS if you need Chinese-first speech synthesis with zero-shot cloning from 3 to 10 seconds of prompt audio, or if you have a polyphone problem that a dynamic controllable dictionary can solve. Do not adopt it if you need a stable, versioned release: the repository shows no retrieved releases, the RL-optimized weights and the 2D Vocos vocoder are both marked as coming soon, and the emotion control that the title advertises depends on a GRPO tree that pulls two extra repositories from GitHub. Verify before committing: whether the ckpt directory contains all five components the README names (Tokenizer, LLM, Flow, Vocoder, Frontend), whether your Python version sits inside the stated 3.10 to 3.12 window, and whether your deployment target is GPU or the Ascend NPU path, since the two use different requirements files and different container images.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. zai-org/GLM-TTS on GitHub