# audio-diffusion: turning mel spectrograms into generated music with diffusers

> teticio/audio-diffusion applies diffusion models to audio by converting sound into mel spectrograms, training an image-style model on those images and converting the result back to sound. It suits researchers and audio developers who want to experiment with latent and conditional music generation, and it now ships an AudioDiffusionPipeline inside the Hugging Face diffusers package, but the pre-trained weights are small and the format changes have broken earlier model checkpoints.

**teticio/audio-diffusion** — Apply diffusion models using the new Hugging Face diffusers package to synthesize music instead of images.

- Repository: https://github.com/teticio/audio-diffusion
- Stars: 794 · Forks: 76
- Language: Jupyter Notebook
- License: GPL-3.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/teticio-audio-diffusion

## Audio treated as an image, then diffused like an image

audio-diffusion is built on a neat representational trick: sound can be drawn. A mel spectrogram is a picture of a sound, with time on one axis and frequency on the other, and once you have that picture you can use image machinery on it. The project description puts it plainly, applying diffusion models to synthesize music instead of images using the Hugging Face diffusers package.

The class that makes the trick work is `Mel` in `mel.py`. According to the documentation it converts a slice of audio into a mel spectrogram of `x_res` x `y_res` and back again, and the resolution is the fidelity knob: the higher it is, the less audio information is lost. The notebook `test_mel.ipynb` is where the conversion is demonstrated.

On top of that representation sits a DDPM, described as De-noising Diffusion Probabilistic Models. A DDPM here is trained on a set of mel spectrograms generated from a directory of audio files, and at inference it synthesizes similar mel spectrograms, which are then converted back into audio. The pipeline is therefore: audio files in, spectrogram images in the middle, audio files out.

That middle step is what makes the project interesting and also where its limits come from. The model never sees raw audio, so it can only reproduce what survives the spectrogram round trip, and it has no notion of song structure. The output is closer to texture and timbre than to a track.

The repository is a Python package rather than a script collection. Top level entries include `audiodiffusion/`, `scripts/`, `config/`, `notebooks/`, `app.py`, `streamlit_app.py`, `pyproject.toml`, `setup.py` and `setup.cfg`, and the Hugging Face Space front matter points at `app_file: app.py` with `sdk: gradio`.

## Latent audio diffusion and the DDIM step reduction

Running a diffusion model over a full 256 x 256 spectrogram is expensive, so the project added a latent variant. The update log for 15/10/2022 records the addition of latent audio diffusion and of DDIM, De-noising Diffusion Implicit Models, with the stated benefit that samples can be generated with far fewer steps, around fifty, than were used in training.

Pre-trained weights for the latent path are published on the Hugging Face Hub: `teticio/latent-audio-diffusion-256` and `teticio/latent-audio-diffusion-ddim-256`. The documentation points out that you can use the pre-trained VAE to train your own latent diffusion models on a different set of audio files. That is the intended entry point for most people: take a trained autoencoder, freeze it, and train the diffusion part on your own audio.

A second capability arrived on 22/10/2022 with a DDIM encoder and the ability to interpolate between audios in latent noise space. The same update removed a constraint that had been there since the beginning: mel spectrograms no longer have to be square, so the vertical frequency resolution and the horizontal time resolution can be set independently. For long audio that matters, because a square spectrogram forces a trade between duration and frequency detail.

The conditional variant came later. The update dated 25/12/2022 states that models can now be trained conditional on an encoding, of text or audio for example, and the model table lists `teticio/conditional-latent-audio-diffusion-512` trained on a 512 resolution dataset.

Generation can also be conditioned on previous audio. The update of 27/9/2022 describes generating audio based on a previous one, which the documentation says can be used to make variations of the same audio or to remix a track, with examples in `test_model.ipynb`.

## Masking lets you stitch samples together like out-painting

One of the more useful mechanics in the project is input masking, recorded in the update dated 4/10/2022 as the ability to mask parts of the input audio during generation so several samples can be stitched together, an approach the documentation likens to out-painting.

This matters because diffusion samples are short and independent by default. Without masking, generating a second snippet gives you a second unrelated snippet, and concatenating them produces an audible seam. With a masked region, the model treats the known parts of the previous sample as context and fills only the gap, so a loop can be extended or a track can be built up in stages while keeping continuity.

The same idea shows up in the sample recording. The README embeds an automatically generated loop and points to a SoundCloud set of loops for more examples, which suggests that short continuous segments are the practical unit of output here rather than complete pieces.

Two more updates round out the feature set. On 2/12/2022 Mel was added to the pipeline and the pre-trained models were changed to save the Mel configuration, which the documentation states makes them no longer compatible with previous versions of the repository. It also notes that migrating previously trained models to the new format is relatively straightforward, and points to the `teticio/audio-diffusion-256` model on the Hub for that purpose.

On 5/12/2022 the project moved into the Hugging Face diffusers package itself: `AudioDiffusionPipeline` was migrated there, which the author describes as making it easier for others to use and contribute.

## Building the spectrogram dataset and pushing it to the Hub

Training starts with turning a folder of audio files into spectrogram images, because the model never trains on audio directly. The project can be installed from GitHub, which includes the training scripts:

```bash
git clone https://github.com/teticio/audio-diffusion.git
cd audio-diffusion
pip install .
```

Or installed from PyPI under the package name `audiodiffusion`:

```bash
pip install audiodiffusion
```

The GitHub route is the one to take if you intend to train, since the scripts directory lives in the repository rather than in the published package.

With the code in place, the dataset is generated by `scripts/audio_to_images.py`. The documentation notes that training at 64x64 mel resolution can be run on a single commercial grade GPU, for example an RTX 2080 Ti, and that `hop_length` should be set to 1024 for better results:

```bash
python scripts/audio_to_images.py \
--resolution 64,64 \
--hop_length 1024 \
--input_dir path-to-audio-files \
--output_dir path-to-output-data
```

For a full resolution dataset the same script writes to a directory and pushes to the Hub, which requires you to be authenticated with `huggingface-cli login` first:

```bash
python scripts/audio_to_images.py \
--resolution 256 \
--input_dir path-to-audio-files \
--output_dir data/audio-diffusion-256 \
--push_to_hub teticio
```

Note the difference between the two invocations. The first passes `--resolution 64,64` as two values, which matches the earlier point about independent horizontal and vertical resolution. The second passes a single `--resolution 256`, which is a square spectrogram.

## Pre-trained checkpoints are small and each covers one narrow dataset

The Hub model table lists seven checkpoints with the dataset each was trained on, and the datasets are as important as the models. `teticio/audio-diffusion-256` was trained on what the table describes as the author's liked Spotify playlist. `teticio/audio-diffusion-breaks-256` was trained on samples that have been used in music, sourced from WhoSampled and YouTube. `teticio/audio-diffusion-instrumental-hiphop-256` covers instrumental hip hop.

The remainder are variants on the same training data rather than new genres. `teticio/audio-diffusion-ddim-256` is a De-noising Diffusion Implicit Model trained on the audio-diffusion-256 dataset, and both latent checkpoints share that same dataset with the 256 entry. The conditional model is the outlier, trained on a separate 512 resolution dataset.

Two practical consequences follow. First, every published checkpoint is small in scope, a 256 pixel spectrogram of short segments trained on a few hundred tracks at most, so the sound you get is a texture of that source audio rather than a general purpose music model. Second, the datasets are published on the Hub as well as the models, which means you can inspect the exact audio a checkpoint learned from and can build a comparable dataset for audio you own.

The licence situation is worth separating from the weights. The repository itself is GPL-3.0, and the Space front matter declares the same licence. That covers the code in `audiodiffusion/`, the scripts and the apps. The documentation does not state a licence for the published model weights or the datasets, so anyone planning to ship generated audio commercially should treat that as an open question rather than assume the repository licence transfers.

## Running it as a Gradio app, a Streamlit app, or a notebook

There are three ways to use the project without writing your own interface, and all three are present in the repository. The Hugging Face Space configuration names `app_file: app.py` with `sdk: gradio` and `sdk_version: 3.1.4`, and the README links an Open in Colab badge pointing at `notebooks/gradio_app.ipynb`. A `streamlit_app.py` sits at the top level next to `app.py`.

The notebooks are the lowest friction route for experimentation. `test_model.ipynb` is where you play with pre-trained models, `test_mel.ipynb` demonstrates the spectrogram conversion in both directions, and `gradio_app.ipynb` is the notebook version of the Space. A published Space at `teticio/audio-diffusion` on the Hugging Face Hub lets you try the model in a browser with no local setup at all.

The requirements file shows what the whole stack costs you in dependencies: `torch`, `numpy`, `Pillow`, `diffusers>=0.40.0`, `huggingface_hub>=1.0`, `librosa`, `datasets>=4.0.0`, `gradio`, `streamlit`, `tensorboard`, `accelerate>=1.10.0`, `torchvision`, `transformers`, `requests` and `ipykernel`. That is a heavy install for a research repository, with a GPU expected for anything beyond loading a checkpoint. The version floors on `diffusers` and `huggingface_hub` are recent, which tells you the project was updated to work against current releases of the Hub libraries even though the tagged releases are older.

There is also a `config/` directory and a `requirements-lock.txt` alongside the looser `requirements.txt`. The lock file is the one to install from if you want the exact versions the project was tested with, and the plain requirements file is what will pick up newer releases.

The primary language of the repository is listed as Jupyter Notebook, and that is an accurate signal about what this project is: a set of experiments and notebooks with a reusable package layer underneath, not a finished tool with a stable command line interface.

## Maintenance state and the cost of a format change

The repository is not archived and the last push was on 2026-09-06, so the code is receiving work. The tagged releases tell a slower story: v1.5.7 from 2024-09-25, v1.5.6 from 2023-05-28 and v1.5.5 from 2023-05-23. The gap between a recent push and a 2024 tag means the released versions on PyPI lag the working tree, so anyone installing `audiodiffusion` from PyPI gets a snapshot that may predate the current `diffusers` requirements.

The most consequential upgrade cost is documented in the project's own history. The 2/12/2022 update changed the pre-trained models so they save the Mel configuration, and states plainly that they are no longer compatible with previous versions of the repository. Anyone holding a checkpoint trained before that change has to migrate it, and the documentation points to the `teticio/audio-diffusion-256` model as the reference for the new format.

The plugin form of the project has also moved. `AudioDiffusionPipeline` now lives in the Hugging Face diffusers package rather than only in this repository, which means the diffusers version you install determines which API you write against. The requirements floor of `diffusers>=0.40.0` is the practical constraint, and pinning higher than that floor to a version whose pipeline signature differs from the one your notes were written against is the failure mode to watch for.

The GPL-3.0 licence is the other thing to settle early. Linking this code into a hosted service or a closed product carries obligations that MIT or Apache licensing would not, and no exception is documented in the repository. This is a statement about the licence terms, not legal advice, but it is the kind of question that has to be answered before the code goes into a pipeline rather than a notebook.

## Conclusion

Adopt audio-diffusion if you are doing research on diffusion-based audio generation and can live with a 256 resolution spectrogram pipeline and small pre-trained checkpoints that were trained on one playlist. Do not adopt it as a music production tool or if you need long, high fidelity, structured audio, and verify first that the diffusers version in your environment matches the AudioDiffusionPipeline API and that any checkpoint you want still uses the current mel format, since models saved before the Mel config change are documented as incompatible.

## FAQ

### What does audio-diffusion use to turn audio into something a diffusion model can learn?

It converts audio into mel spectrograms with the `Mel` class in `mel.py`, which works in both directions between an audio slice and an `x_res` x `y_res` spectrogram. A higher resolution loses less audio information.

### How do I install audio-diffusion and train my own model?

Clone the repository and run `pip install .`, which includes the training scripts, or install the published package with `pip install audiodiffusion`. Then generate a dataset with `scripts/audio_to_images.py` before training.

### Do the older pre-trained audio-diffusion models still work?

The update dated 2/12/2022 states that the pre-trained models now save the Mel configuration and are no longer compatible with previous versions of the repository. The documentation says migrating previously trained models to the new format is relatively straightforward.

### What are the main pre-trained models and what were they trained on?

The table lists seven checkpoints, including `teticio/audio-diffusion-256` trained on a liked Spotify playlist, `teticio/audio-diffusion-breaks-256` trained on samples sourced from WhoSampled and YouTube, `teticio/audio-diffusion-instrumental-hiphop-256`, DDIM and latent variants, and a conditional 512 model.

### Can audio-diffusion generate audio that continues an existing sample?

Yes. The update of 4/10/2022 describes masking parts of the input audio during generation so several samples can be stitched together, and the update of 27/9/2022 describes generating audio based on a previous one to make variations or remixes.

### Is audio-diffusion still being maintained?

The repository is not archived and the last push was on 2026-09-06. The tagged releases are older, with v1.5.7 published on 2024-09-25, so the PyPI package may lag the working tree.

## Sources

- [Issues](https://github.com/teticio/audio-diffusion/issues)
- [License: GPL-3.0](https://github.com/teticio/audio-diffusion/blob/main/LICENSE)
- [README](https://github.com/teticio/audio-diffusion/blob/main/README.md)
- [Releases](https://github.com/teticio/audio-diffusion/releases)
- [teticio/audio-diffusion on GitHub](https://github.com/teticio/audio-diffusion)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/teticio-audio-diffusion