Audio8_TTS Preview: a 0.6B zero-shot cloning TTS model and how to run it

What Audio8_TTS solves, and who the 11-language limit excludes

Open-weights text-to-speech usually forces a choice. Small models are fast but sound flat and cannot clone a voice; large models clone well but need a GPU you may not have. Audio8_TTS Preview sits in the middle by design: the README describes it as "a 0.6B-parameter multilingual text-to-speech model with zero-shot voice cloning." The main model is 601,159,424 parameters excluding the codec, and the checkpoint bundles its own neural codec, so reference encoding and waveform decoding do not require a second model download.

The intended user is someone building speech output who wants to supply a short reference recording and get that voice back, without a hosted API. The README's own framing is scale: "How small a zero-shot cloning TTS can be?" That is a size argument, not a quality argument, and the documentation does not publish a comparison against any other model.

The constraint that matters most is language. The Preview checkpoint is documented as performing best in 11 languages: Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish and Spanish. The README states plainly that coverage is intentionally limited in this release and that Chinese dialect support will come later. If your product ships in Portuguese, Hindi or Arabic, this checkpoint is the wrong tool today, and no amount of prompt engineering fixes a language the model was not trained on.

A second boundary is input length. The README advises keeping each input within 150 characters and splitting longer text, because longer input may reduce generation quality. That is a real design limit, not a suggestion about style. Anyone generating audiobooks or long-form narration has to write their own segmentation and stitching layer, and the repository does not ship one.

DualAR: one semantic token per frame, then ten codebooks

The architecture is a DualAR design, which the README says is inspired by Fish Audio S2 Pro. Two autoregressive transformers run in sequence rather than in parallel.

The slow AR transformer is 24 layers wide, with model width 896, 14 attention heads and 2 KV heads. For each audio frame it predicts a single semantic token. The fast AR transformer is much smaller, 4 layers at the same width and head counts, and its job is to expand that one semantic token into the frame's acoustic content: 10 codebooks with 4,096 entries each. The fast branch is conditioned on the slow hidden state and on the codebooks it has already produced. Both branches use static KV caches during generation.

The codec side sets the timing. It runs at 44.1 kHz with 2,048 samples per model frame, which the README works out to roughly 21.5 frames per second. Context is capped at 2,048 packed text and audio positions. That cap is shared between the prompt text and the reference audio, so a long reference recording eats into the budget available for the text you actually want spoken. It also explains the 150-character advice: the ceiling is positions, not words.

The 0.1B variant changes one piece. `Audio8-TTS-Preview-0.1b` replaces the pure-attention slow backbone with a Falcon-H1 hybrid combining Mamba 2 SSM layers and attention. The SGLang adapter detects this from `config.json`, looking for `slow_backbone: falcon_h1` or a `mamba_d_ssm` field, and switches to an eager hybrid path. That detection is convenient but also means the two checkpoints do not share one inference code path, so a bug fixed for one is not automatically fixed for the other.

Installing Audio8_TTS and cloning a voice on the first run

The README asks for Python 3.10 or newer and recommends a CUDA-capable GPU. `pyproject.toml` sets the Ruff target to py310 and a line length of 100, which is consistent with that floor. Create an isolated environment first, then install the pinned requirements.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

`requirements.txt` pulls `torch>=2.5.0`, `torchaudio>=2.5.0`, `transformers>=4.57.0,<5`, `numpy>=1.26`, `soundfile>=0.12`, `safetensors>=0.4` and `tqdm>=4.66`. Note the upper bound on Transformers: this is not a project that tracks the newest release.

Next, get the weights. The README points to the Hugging Face repository Audio8-TTS-Preview-0.6b and says to place the checkpoint in the repository's `model/` directory, with an expected local path of `model/audio8_tts_0_6B_preview/`. Every command also accepts a Hugging Face model ID through `--model`, so you can skip the manual placement if you would rather let the download happen at run time.

With weights in place, zero-shot cloning is one command. The reference transcript should match what is actually spoken in the reference audio, which is the part people get wrong most often.

python audio8_tts_infer.py \
  --text "Welcome to audio8_tts." \
  --reference-audio examples/reference.wav \
  --reference-text "Transcript of the reference recording." \
  --output outputs/clone.wav

The same script runs without a reference voice, which is useful for checking that the install works before you hunt for a clean recording.

python audio8_tts_infer.py \
  --text "This utterance does not use a reference voice." \
  --output outputs/no_reference.wav

For more than a handful of lines, batch mode reads a JSONL manifest where each line is an independent object and relative audio paths resolve from the manifest's directory.

{"id":"sample_001","text":"Target text","reference_audio":"audio/ref.wav","reference_text":"Reference transcript"}
{"id":"sample_002","text":"Text without a reference voice"}

python audio8_tts_infer.py \
  --input-jsonl data/prompts.jsonl \
  --output-dir outputs/batch \
  --batch-size 2

The batch run writes `manifest.jsonl` and `failures.jsonl`. Existing WAV files are skipped unless you pass `--overwrite`, so a rerun after a crash resumes rather than redoes. Run `python audio8_tts_infer.py --help` to see the sampling and code-saving options; the README defers to that help text rather than listing them.

The CPU ONNX path, and why it is a separate deployment

The `onnx_runtime/` directory holds a standalone CPU deployment that does not use PyTorch or Transformers at all. It runs weight-only INT4 Slow and Fast AR models with FP16 activations and KV caches, plus an FP16 codec. The README lists CLI inference, a local web and HTTP service, streaming PCM output, and reference-voice registration.

The memory figures in the README are specific: the online sessions use about 1 GiB of memory in the tested Apple M2 setup. It also describes a sequencing trick worth knowing about, where the online sessions are released before the codec encoder is loaded during voice registration, to keep peak memory down. That is a deliberate design choice, and it tells you the author was working against a real memory ceiling rather than guessing.

Two things to weigh before choosing this path. First, it is a different code path from the PyTorch inference script, so quality and behaviour can diverge; the README does not claim they match. Second, the memory number is tied to one machine. It is a data point, not a specification, and there is no published figure for x86 CPUs or for the 0.1B INT8 variant in `onnx_runtime_0_1b_int8/`. Download the ONNX model from the Audio8-TTS-Preview-0.6B-ONNX-INT4 repository and follow `onnx_runtime/README.md`; the top-level README does not repeat those steps.

SGLang Omni serving is the least stable part of the stack

The `sglang_omni/` adapter turns the model into an OpenAI-compatible service with SGLang paged attention, dynamic batching, a fixed KV cache for the fast codebook decoder, reference-audio encoding and waveform decoding. It installs as an independent `audio8_tts` model plugin, which the README says does not overwrite SGLang Omni core files.

The compatibility table is the important part, and it is unusually honest. The adapter uses internal SGLang Omni interfaces, so the README tells you to deploy with the tested revision instead of the latest `main` branch. The tested set is SGLang Omni commit `68a572348837f7b004857b4b07993c20ade4c017` (version `0.1.0`), SGLang `0.5.8`, PyTorch `2.9.1+cu128`, Transformers `4.57.1` and BF16 precision.

Read that as a maintenance warning. Depending on internal interfaces means an upstream refactor can break this adapter without any change on the Audio8 side, and the pinned Transformers `4.57.1` here is stricter than the `>=4.57.0,<5` range in `requirements.txt`. If you need a serving endpoint that survives routine dependency upgrades, this is the wrong layer to build on right now. If you can freeze the whole environment, it is a reasonable way to get batching and an OpenAI-shaped API in front of the model.

Where Audio8_TTS is the wrong choice, and what to use instead

The clearest failure mode is a language outside the 11 listed. The README calls the coverage intentionally limited, and no configuration flag expands it. A second failure mode is long input: past roughly 150 characters the README warns that quality may drop, and the 2,048-position context is shared with reference audio, so a long reference makes the effective text budget smaller still. A third is voice cloning from a mismatched transcript. The README states the reference transcript should match the spoken content, and it does not document any fallback when it does not.

For the multilingual case, the honest alternative is a hosted API such as ElevenLabs or OpenAI's speech endpoint. The difference is not just quality: those services own the model, the scaling and the language coverage, and you trade a per-character bill and a network round trip for not running a GPU. Audio8_TTS gives you the weights under Apache-2.0 and lets you run offline, which matters when the audio cannot leave your infrastructure.

For the long-form case, the alternative is a non-autoregressive TTS system built for paragraph input, where the model handles segmentation internally. Audio8_TTS does not; you would write the splitter and the crossfade yourself, and the seams are your problem.

For the serving case, the alternative is the plain PyTorch script behind your own queue and worker pool. You lose SGLang's paged attention and dynamic batching, but you also stop depending on a pinned internal interface. Given that the README explicitly warns against the latest `main` branch, that trade is worth taking seriously.

Licence, training code and what a fork actually costs

The repository is Apache-2.0, with a `NOTICE` file at the top level. Apache-2.0 permits commercial use and modification and includes a patent grant, but it also requires that you keep the licence and notice files and state what you changed. The README does not discuss model-weight licensing separately from the code, and it does not address voice-cloning consent, which is a policy question your legal team will have to answer rather than a licence question. Nothing here is legal advice.

The maintenance picture is bounded. The last push to the default branch was on 2026-09-02, and the repository is not archived. There are no retrieved releases, so there is no versioned changelog to diff against and no tagged rollback point. Upgrades arrive as commits on `master`, which means the practical cost of tracking the project is reading diffs rather than bumping a version pin.

The repository also ships an independent SFT pipeline: `audio8_tts_prepare.py`, `audio8_tts_data.py`, `audio8_tts_sft.py` and `audio8_tts_sft.sh`, with `requirements-train.txt` kept separate from the inference requirements. That is the part that makes a fork viable for a language the Preview does not cover, but it is also the part the README covers least. Fine-tuning is a real project, not a configuration change, and the documentation gives you the entry points without a worked example.

Editorial conclusion

Adopt Audio8_TTS if you need a 0.6B open-weights cloning model you can run locally, and start with the PyTorch path because it is the one the README documents end to end. Do not adopt it if you need a stable production serving stack: the SGLang Omni adapter depends on internal interfaces and the README pins it to a specific commit. Before committing, verify the checkpoint actually lands in model/audio8_tts_0_6B_preview/, test your language against the 11 listed ones, and time a full generation on your own GPU, since the README gives no latency or real-time-factor numbers.

Frequently asked questions

Official sources

1. Edge0-AI/Audio8_TTS on GitHub
2. Issues
3. License: Apache-2.0
4. README