Large-Audio-Models is a reading list for speech, singing and music models

What the repository actually contains

The repository states its purpose in one line: it keeps track of something big in the audio domain, including speech, singing and music. The top level of the repository has two entries, README.md and a file called .tags, and there is no source code, no build system and no package manifest. What you get is a curated list. Each entry is a paper with a title, a year, an author list and links, and the format is consistent enough to scan: a bold title with the year in parentheses, the names, then a PDF link and sometimes a code link to the project repository. The README has a table of contents with four sections, which means the list is organised by the kind of model rather than by year, and that a reader who wants speech interaction and a reader who wants text-to-music are looking in different places.

Spoken language models, the newest section

The spoken language models section is the only one that mixes 2023 and 2024 entries, and it holds the four items the list treats as the current generation of models that both hear and speak. Moshi is described as a speech-text foundation model for real-time dialogue from 2024.9, by Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour, and it is the only entry in the whole list with a code link pointing at a lab repository rather than a paper repository. LLaMA-Omni covers speech interaction with large language models, also 2024.9, by Qingkai Fang and others, and it carries a code link. Mini-Omni, from 2024.8, is described as language models that can hear and talk while thinking in streaming. SpeechGPT, from 2023.5, is the older entry by Dong Zhang and others. The progression from a standalone speech model to a system integrated with a text model is visible in the order of these titles.

Prompt-based audio synthesis, and where diffusion enters

The longest section, prompt-based audio synthesis, is almost entirely 2023 work with a couple of 2022 entries at the bottom, and the entries fall into clear families. Music generation from text: M2UGen on multi-modal music understanding and generation, TANGO using an instruction tuned language model with a latent diffusion model, Mustango for controllable text-to-music, ERNIE-Music for text-to-waveform generation, Noise2Music for text-conditioned music generation, MusicLM for generating music from text, Moûsai for long-context latent diffusion, and the Simple and Controllable Music Generation and Efficient Neural Music Generation papers. Text-to-speech and speech editing: NaturalSpeech 2 for zero-shot speech and singing synthesis, AUDIT for audio editing by following instructions, Spear-TTS for high-fidelity text-to-speech with minimal supervision, VALL-E and VALL-E X for cross-lingual neural codec language modelling, PromptTTS for controllable text-to-speech with text descriptions, InstructTTS for expressive TTS in a discrete latent space, and FoundationTTS for text-to-speech customised for ASR. Sound generation of other kinds covers Diffsound, Physics-Driven Diffusion Models for impact sounds from video, and ArchiSound. The recurring pattern is that a language model produces a semantic description that a diffusion model turns into a waveform.

Audio language models, the unifying idea

The audio language models section gathers work that treats audio the way a text model treats text. AudioLM is described as a language modeling approach to audio generation, from 2022, by Zalán Borsos and others. AudioPaLM is a large language model that can speak and listen, and Pengi is an audio language model for audio tasks, both from 2023. SpeechTokenizer is a unified speech tokenizer for speech large language models. SoundStorm is efficient parallel audio generation, and AudioGPT handles understanding and generating speech, music, sound and talking heads. Qwen-Audio advances universal audio understanding through unified large-scale audio-language models, and UniAudio heads toward universal audio generation. FunAudioLLM, the most recent entry at 2024, is a voice understanding and generation foundation model for natural interaction between humans and language models, and it is also one of the entries with a code link, pointing at the SenseVoice repository.

Self-supervised audio models, the layer underneath

The fourth section covers audio self-supervised and unsupervised learning models, and it is where the list reaches furthest back. vq-wav2vec from 2019 is self-supervised learning of discrete speech representations, and wav2vec 2.0 from 2020 is a framework for self-supervised learning of speech representations, both by Alexei Baevski and others. W2v-BERT is the entry that combines contrastive learning with masked language modelling, and the list continues from there. This section matters as context for the others, because the tokenizers and the learned representations that the later language models consume come from this line of work. A reader who has only read the 2023 generation papers will not have the background for why a discrete speech token vocabulary is the right interface, and this section supplies it. It is also the part of the list where the boundaries of the collection are most visible, since the README cuts off mid-entry in the W2v-BERT line.

Why a list like this is worth keeping

The value of a bibliography is not the entries but the selection, and the selection here is organised around a specific question: how do you make a model that treats sound as a language. The order of the sections is the argument, starting with self-supervised representation learning, moving to audio language models that make that representation generative, and then to the synthesis work where a language model drives a diffusion model, and finally to the spoken language models where the whole thing becomes a conversation. A reader who follows the four sections in that order gets a coherent path through a field that is otherwise a pile of unrelated papers, and each entry is a link to the original rather than a summary that might have lost the caveats. For a person deciding which paper to read first in an unfamiliar area, that is the difference between a directory and a guide.

Where a reading list falls short

Three limits. The first is currency: the list has no status markers, no stars and no indication of which entries are still competitive, so a reader cannot tell a paper that is still the state of the art from one that has been overtaken, and the last push was on 2026-07-03, which dates the collection. The second is depth: an entry is a title, a year, an author list and a link, so the list cannot tell you what a model does, what it was evaluated on, or why the author thought it mattered, and anyone wanting that has to read the PDF anyway. The third is coverage: the four sections are a particular view of the audio field, and a paper that does not fit the taxonomy is simply absent, with no way to tell an omission from an oversight. Against a conference proceedings listing, the difference in approach is curation rather than exhaustiveness, and the price is that the list is a point of view that will age.

Editorial conclusion

Large-Audio-Models is a good fit for a researcher who wants a map of the audio model literature rather than a tool, and for a reader who wants to find the paper that introduced a specific idea and then read the original rather than a summary. It is a poor fit if you need running code, because the repository holds a README and a tags file and no source, and it is a poor fit if you need a current list, because the entries stop at the papers listed and the last push was on 2026-07-03. Check three things before you rely on it: that the paper you need is in one of the four sections rather than one you assumed was covered, that the entry has a code link before you plan to run anything, since many entries carry only a PDF, and that the list's own date range is acceptable to you, because the entries are grouped by kind of model and are not marked as current or superseded.

Frequently asked questions

Official sources

1. Issues
2. liusongxiang/Large-Audio-Models on GitHub
3. README