What is Text-to-speech (TTS)?

How a voice gets made

The pipeline has three parts, and the interesting engineering moved between them over the years.

The text frontend. Raw text is not speech input. Numbers become words, abbreviations are expanded, and the result is turned into phonemes, with a pronunciation lexicon overriding the default for names and jargon. A second pass adds linguistic features: which syllables are stressed, how long they last, where the intonation rises. This stage is unglamorous and it is where most embarrassing failures live, because everything downstream faithfully speaks whatever it was given.

The acoustic model. The older design was two models: one predicting how long each phoneme lasts and with what pitch, another turning that into a spectrogram. The current design is a single autoregressive model that consumes the phoneme sequence plus the linguistic features and emits acoustic tokens, trained on a large corpus of recorded speech. fishaudio/fish-speech is one of these: our analysis describes it as a multilingual system built around a 4-billion-parameter Dual-Autoregressive architecture trained on more than ten million hours of audio.

The vocoder. Finally a model turns the predicted spectrogram into a waveform. This is where sample rate and audio quality are decided, and it is the part that has to run fast enough to be real time.

One axis is still being argued over. Most systems insert a discrete tokenizer between the acoustic model and the vocoder, which quantises the spectrogram into a fixed set of codes and makes generation easy but lossy. OpenBMB/VoxCPM is the interesting exception: our analysis describes VoxCPM2 as a 2-billion-parameter model that generates speech without discrete tokenization, using a diffusion autoregressive architecture, supporting thirty languages, real-time voice cloning and 48kHz output, under an Apache-2.0 licence and a single pip install.

Why cloning became possible

Cloning is not a separate feature bolted on. It falls out of the same architecture.

When the acoustic model consumes a phoneme sequence, the voice has to come from somewhere. The older way was a catalogue: the model was trained on a set of named speakers and selected one at inference time, so the set of voices was fixed when training finished. The current way is to prepend the acoustic representation of a short reference clip to the phoneme sequence and let the model continue in that voice. Any voice becomes available without retraining, and the training corpus is not a curated speaker set but whatever speech exists.

That produces the two tiers everyone uses. Zero-shot cloning is the reference clip alone, with no training at all, and it is what a web demo does. Few-shot cloning is an actual fine-tune on a few minutes of one voice, and it is what a production voice needs. RVC-Boss/GPT-SoVITS makes the structure explicit rather than implicit, and our analysis of it notes the real cost is data preparation rather than training, which is true of every few-shot voice: the minute of clean, correctly licensed speech is the expensive part, and the training run is the cheap part.

Cloning also split into two schools. myshell-ai/OpenVoice separates tone colour from style, per our analysis, so you clone the timbre from a reference and then control emotion, accent, rhythm and pauses as a separate layer. index-tts/index-tts takes the other direction and attacks pronunciation: our analysis describes it cloning a voice from one reference clip and adding pinyin, CMU phoneme and Kana pronunciation control plus a duration factor for speaking speed, which is a direct answer to the most common complaint about generated speech.

When you need it, and when you should not

The case is strongest where the voice is yours and the volume is real.

It fits offline and private audio. Narration, course material, internal tools, anything where a recording cannot be uploaded. DrewThomasson/ebook2audiobook is the packaged version: our analysis describes it turning EPUB, PDF and other e-book files into chaptered audiobooks using XTTSv2 and other engines, with optional voice cloning and a Gradio interface, as a local tool that is friendly to Docker.

It fits volume. A per-character hosted bill over a large archive is a machine you already own.

It fits small devices. KittenML/KittenTTS is the extreme case: our analysis describes it as an Apache-2.0 Python library that runs ONNX models of 15M to 80M parameters without a GPU, which is a different order of magnitude from the multi-billion-parameter systems and is the right shape for something embedded or on a laptop.

It fits when pronunciation is the problem. A model that misreads your product names will misread them every time, and the fix is a pronunciation lexicon or the phoneme-level control that IndexTTS exposes.

It does not fit when you need the best available quality and no GPU. Customisation and most studio tooling want real hardware, and our analysis of debpalash/OmniVoice-Studio is direct that its desktop studio, which bundles sixteen text-to-speech engines and eleven speech-recognition engines behind one interface, asks for a real GPU and a tolerance for beta software in return.

It does not fit when you need twenty reliable voices and have not budgeted for data preparation, because each voice is a data project and not a setting. And it does not fit when a hosted catalogue is genuinely what you want, where the voices are already recorded by people whose day job is that.

Where it goes wrong

Six, in the order they get discovered.

The text frontend. An account number read as a sequence of digits, an abbreviation expanded wrongly, a surname mangled. The model is not at fault and there is no error, only a confident mispronunciation. A pronunciation lexicon for your domain is the cheapest fix in this list.

The int8 build. Quantising a small model to run faster is a trade, and our analysis of KittenTTS notes that it is a developer preview and that the int8 build is the one users have complained about. The same trade exists in larger systems, and it is worth listening to the audio rather than the size.

Artefacts from generating too fast. Real-time budgets compress the model, and the audible results are clipped words, flattened prosody, or a cut-off at the end of a chunk. A model that sounds fine offline can be unusable streaming.

The reference clip decides the voice. Cloning a clip with room noise, a thin telephone band or an unusual emotion gives you that voice, and there is no second attempt. Recording the reference properly is worth more than any model choice.

Version drift. A cloned voice is not the same voice in the next model version, so a pipeline that regenerates narration with an upgraded checkpoint will not sound identical. Pin the checkpoint if consistency matters.

Weights and code. The pattern is consistent across this ecosystem: the repository licence and the licence on the weights are separate questions, and a permissive repository says nothing about the second. Read both before you ship.

How the projects divide

Six groups, and the group tells you whether you are running a model or a studio.

The library. coqui-ai/TTS is the long-standing Python toolkit with released pretrained models, training recipes and an API, and our analysis records its last push as 2023-12-12 and describes it as a stable codebase rather than a moving target, which is the right way to think about adopting it and the wrong way to think about waiting for features.

The two-stage cloner. RVC-Boss/GPT-SoVITS is the reference arrangement for zero-shot and few-shot synthesis in one interface, and the reason it is popular is that the two stages are separately controllable, which is what makes fine-tuning a specific voice practical without a research team.

The acoustic models. fishaudio/fish-speech with its 4-billion-parameter Dual-Autoregressive architecture, and OpenBMB/VoxCPM with its tokenizer-free diffusion autoregressive approach. Our analysis of fish-speech also records that the S2 Pro model scores best overall on the Seed-TTS Eval word error rate in both Chinese and English and supports fine-grained emotion control through in-text tags, which is a different control surface from phoneme-level control and worth trying before you build a pronunciation lexicon.

The controllable cloners. myshell-ai/OpenVoice for tone colour separated from style, and index-tts/index-tts for explicit phoneme-level pronunciation control and a speed factor.

The studio and the product. debpalash/OmniVoice-Studio bundles sixteen synthesis engines and eleven recognition engines under one desktop interface, and DrewThomasson/ebook2audiobook wraps the whole thing for one job: turning a book into chapters of audio.

The framework. NVIDIA-NeMo/Speech is the one to read when you are training rather than deploying, with our analysis describing training infrastructure, checkpoints on HuggingFace and NVIDIA GPU containers, and recording that it narrowed its scope to audio and speech in 2026 after splitting from the broader NeMo framework, which matters if you are choosing a dependency to build on.

In practice

Text-to-speech is a text frontend, an acoustic model and a vocoder, and the shift that made voice cloning ordinary was changing the acoustic model from a fixed catalogue of speakers to one conditioned on a reference clip. Choose by where it runs and what you need to control: KittenML/KittenTTS when it has to fit somewhere without a GPU, coqui-ai/TTS as a library you extend yourself, RVC-Boss/GPT-SoVITS when you need a specific voice trained properly, OpenBMB/VoxCPM or fishaudio/fish-speech when quality is the constraint, and index-tts/index-tts or myshell-ai/OpenVoice when pronunciation and style are the things you have to steer. Before any of it, record the reference clip properly and build a pronunciation lexicon, because those two things fix more output problems than any model will.