fish-diffusion: a diffusion voice framework split into readable training tools

Three tasks, one diffusion model, and an explicitly decoupled codebase

Fish Diffusion describes itself as an easy to understand TTS, SVS and SVC training framework, and the three letters are the three voice tasks: text to speech, singing voice synthesis, and singing voice conversion. The approach is to use a diffusion model to solve each of them, and the repository compares itself with the original diffsvc project it grew out of.

The stated advantages are concrete. Multi-speaker training is supported, the code structure is simpler to understand, all modules are decoupled, the 44.1kHz DiffSinger community vocoder is supported as an alternative, and multi-machine multi-device training is supported along with half-precision training, with the claim that this saves training speed and memory.

The decoupled claim is visible in the tree. The top-level entries are fish_diffusion/ for the library itself, tools/ for the command line entry points, configs/ for experiment configurations, dictionaries/ for what look like phoneme or symbol tables, tests/, docs/, notebooks/, and a run.sh. The entry points in tools/ are separate scripts rather than subcommands of one tool, which is exactly what makes the pipeline readable: you can read one file and know what it does.

Environment setup assumes conda, Python 3.10 and a pinned CUDA 11.8 PyTorch

The installation instructions are short and assume a conda environment on Python 3.10. The three steps are PyTorch, the dependency manager, and the project sync:

# Install PyTorch related core dependencies, skip if installed
# Reference: https://pytorch.org/get-started/locally/
conda install "pytorch>=2.0.0" "torchvision>=0.15.0" "torchaudio>=2.0.0" pytorch-cuda=11.8 -c pytorch -c nvidia

# Install PDM dependency management tool, skip if installed
# Reference: https://pdm.fming.dev/latest/
curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py | python3 -

# Install the project dependencies
pdm sync

PDM is the dependency manager, and pdm sync installs from the committed pdm.lock. The pyproject.toml requires Python 3.10 or newer, below 4.0, and names the package fish-diffusion at version 2.2.0.

There is a version inconsistency worth knowing about before you install. The pyproject dependency ranges are modern, with pytorch-lightning at 2.0.2 or above, transformers at 4.33.3 or above, diffusers at 0.21.2 or above and deepspeed at 0.10.3 or above. The committed requirements.txt is much older, pinning pytorch-lightning 1.9.4, mmengine 0.4.0 and librosa 0.9.1, while pyproject asks for mmengine 0.7.3 and librosa 0.9.1 or above. The pyproject.toml also has an autoexport rule that writes requirements.txt from the project metadata, so the committed file is a generated snapshot rather than the source of truth. Install with pdm sync, and if you fall back to requirements.txt you are installing a different dependency set.

The vocoder is a separate download with a non-commercial licence

Fish Diffusion does not ship a vocoder. It requires the FishAudio NSF-HiFiGAN vocoder to generate audio, and the documentation gives two ways to get it. The automatic route is a script in the tools directory:

python tools/download_nsf_hifigan.py

That script can accept the licence on your behalf, with the --agree-license flag:

python tools/download_nsf_hifigan.py --agree-license

The flag is not a formality. The script will otherwise stop, because the vocoder is distributed under CC BY-NC-SA 4.0, a licence with a non-commercial condition. The manual route is to download nsf_hifigan-stable-v1.zip from the project's release page and copy the nsf_hifigan folder into the checkpoints directory, creating it if it does not exist. The documentation also points at ContentVec, a content feature extractor, with a manual download for anyone who prefers to place the checkpoint there themselves.

This is the first hard boundary of the project. A framework that produces research audio and a framework that produces audio you can ship are separated here by a licence flag, and the flag is part of the setup step rather than something you discover later.

Datasets are plain directories of wav files, and multi-speaker is a folder convention

The dataset format is deliberately dull, which is a virtue in a training framework. You put wav files into a dataset directory with train and valid subdirectories:

dataset
├───train
│   ├───xxx1-xxx1.wav
│   ├───...
│   ├───Lxx-0xx8.wav
│   └───speaker0 (Subdirectory is also supported)
│       └───xxx1-xxx1.wav
└───valid
    ├───xx2-0xxx2.wav
    ├───...
    └───xxx7-xxx007.wav

The file naming pattern is visible in the example names, and the documentation notes that a speaker subdirectory is also supported inside train, which is how the multi-speaker support is expressed in practice.

Once the wav files are in place, a feature extraction pass turns them into what the model trains on:

# Extract all data features, such as pitch, text features, mel features, etc.
python tools/preprocessing/extract_features.py --config configs/svc_hubert_soft.py --path dataset --clean

The --clean flag means the feature directory is rebuilt rather than merged, and the config argument selects the feature set. The dependency list explains what those features are made of: praat-parselmouth for pitch, openai-whisper for text features, pypinyin and pykakasi for Chinese text handling, and pyworld and torchcrepe for pitch extraction. The transcription and text handling are Chinese-oriented, which tells you the framework was built for Mandarin singing and speech data.

Training is one script with several config files, and the docs warn you about them

The training entry point takes a config file and nothing else, and the difference between single machine and multi-node training is a different config:

# Single machine single card / multi-card training
python tools/diffusion/train.py --config configs/svc_hubert_soft.py
# Multi-node training
python tools/diffusion/train.py --config configs/svc_content_vec_multi_node.py

Resuming and fine-tuning use the same script with different flags. Resuming takes a checkpoint file, and fine-tuning takes a pretrained checkpoint with a config chosen for that purpose, and the documentation adds a note that you need to adjust the learning rate scheduler in the config file to warmup_cosine_finetune.

# Resume training
python tools/diffusion/train.py --config configs/svc_hubert_soft.py --resume [checkpoint file]

# Fine-tune the pre-trained model
# Note: You should adjust the learning rate scheduler in the config file to warmup_cosine_finetune
python tools/diffusion/train.py --config configs/svc_cn_hubert_soft_finetune.py --pretrained [checkpoint file]

The config file naming carries real information. svc_ is the singing voice conversion family, cn_ marks a Chinese configuration, hubert_soft and content_vec name the feature extractors, and multi_node marks the distributed setup. The cost of the config-driven design is that the config file is the real interface, and the documentation warns three times in a row that the project is under active development and to back up your config file. That warning is the honest signal: an experiment can break because a config file changed.

Inference is a shell script and a Gradio web interface over the same entry point

Running a trained model follows the same pattern. The inference script takes a config, a checkpoint, an input file and an output file:

# Inference using shell, you can use --help to view more parameters
python tools/diffusion/inference.py --config [config] \
    --checkpoint [checkpoint file] \
    --input [input audio] \
    --output [output audio]

The documentation then describes a Gradio web inference option, with the note that other parameters will use the Gradio defaults. Gradio is in the dependency list, along with FastAPI pinned at 0.88.0, Flask and flask-cors, which tells you the project also has a service-style path. The model is served behind a web interface as well as a command line, and the difference is the parameters you can pass: the shell path exposes the script's own arguments, the Gradio path exposes whatever the interface was written to expose.

For training monitoring, the dependencies include wandb and tensorboard, and pytorch-lightning is the training framework, so runs can be logged and resumed from checkpoints rather than started over.

The container path is unusual and worth understanding before you use it. The Dockerfile is based on python:3.10-bullseye rather than a CUDA image, it installs ffmpeg and other system packages, creates a user named fish with sudo in the group, installs PDM, clones the repository, runs pdm sync, and downloads the NSF-HiFiGAN vocoder with the licence flag already set. The entry point is the SSH daemon, not a training script, so the image is a workspace you connect into rather than a job you run.

The terms of use are the real constraint, not the model

The repository carries a terms of use document that is more demanding than the MIT licence alone suggests, and anyone training a voice model should read it before starting.

Four clauses matter. The first puts responsibility for obtaining authorisation for training datasets entirely on the user, with the project disclaiming responsibility for problems arising from unauthorised datasets. The third requires derivative works, including audio and video, to acknowledge the project and to declare that the content is AI-generated, and to carry the original links for third party material. The second requires derivative works to credit the project and its licence, and to cite the original author and source code when distributing code or results. The fourth states that continued use constitutes agreement to the terms and that neither the project nor its developers will be liable for resulting difficulties.

So the licence and the terms are two different things. MIT governs the code. The terms govern what you may do with the code and with what it produces, and they are aimed squarely at the voice cloning use case, where a model trained on one person's audio can be used to say things they did not say. Combined with the CC BY-NC-SA 4.0 vocoder licence, the practical picture is that the default path through this framework is research and non-commercial use.

One more limit belongs here rather than in a feature list. The newest tagged release is v2.2.1 from 2023-05-22, released after v2.1.0 and v2.0.0, and the version in pyproject.toml is 2.2.0. The last push to the repository was on 2026-10-05, so work continues on main, but the packaged releases stopped in 2023 and the documentation's own warning about backing up config files is a symptom of that gap.

Editorial conclusion

fish-diffusion fits a researcher who wants to understand and modify a diffusion voice training loop rather than call a hosted service, because the training entry point, the feature extraction pass and the vocoder are all separate commands you can run and read. It does not fit a team that needs a maintained production path, since the newest tagged release is v2.2.1 from 2023-05-22 and the project version in pyproject.toml is still 2.2.0, so you are running a framework whose packaging has barely moved in three years even though the last push was on 2026-10-05. Before you invest, check the three things that decide the outcome: that your dataset is one you are authorised to train on, since the terms of use put that responsibility entirely on the user; that the FishAudio NSF-HiFiGAN vocoder licence, CC BY-NC-SA 4.0, is compatible with what you intend to do with the output; and that the pinned requirements still resolve, because the locked requirements.txt and the dependency ranges in pyproject.toml disagree on several versions of the same packages.

Frequently asked questions

Official sources

1. fishaudio/fish-diffusion on GitHub
2. License: MIT
3. Project website
4. README
5. Releases