# AnyGPT: text, speech, images and music as one token stream

> AnyGPT is a multimodal language model that converts every modality into a discrete sequence and trains on them with next token prediction, and it ships base weights, chat weights, inference scripts and the AnyInstruct dataset. The generation is documented as unstable, and the repository names no standard licence.

**OpenMOSS/AnyGPT** — A unified multimodal language model based on discrete sequence modeling

- Repository: https://github.com/OpenMOSS/AnyGPT
- Website: https://junzhan2000.github.io/AnyGPT.github.io/
- Stars: 888 · Forks: 76
- Language: Python
- License: not declared
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/openmoss-anygpt

## Everything becomes a discrete sequence before training starts

AnyGPT is an any-to-any multimodal language model that uses discrete representations for the unified processing of speech, text, images and music. The training scheme converts all modal data into a unified discrete representation and then applies the Next Token Prediction task on a Large Language Model, which is what makes a single model handle all four modalities rather than four models bolted together. The README frames this with a slogan about compression being intelligence: when the quality of the tokenizer is high enough and the perplexity of the language model is low enough, a very large amount of multimodal data on the internet can be compressed into one model, and capabilities emerge that a text-only model does not have. The practical consequence for a user is that the model is not a stack of specialist encoders you can swap out. It is one sequence model with a shared vocabulary, so the modality boundaries live in the tokenizers rather than in the architecture.

## Two checkpoints and a dataset that sits between them

The repository publishes four things, which the README lists as an open-source checklist: the base model, the chat model, the inference code and the instruction dataset. The base model aligns the four modalities and allows conversions between different modalities and text. The chat model was trained on AnyInstruct, a dataset built from various generative models that contains instructions for arbitrary modal interconversion, and it can hold free multimodal conversations where multimodal data is inserted at will. So the base model is the alignment stage and the chat model is the instruction-following stage on top of it. The dataset is what makes the chat model possible, and it is published on Hugging Face as fnlp/AnyInstruct. That combination of weights plus data plus code is more complete than most research releases, and it also means the reproduction path does not require you to build your own training corpus.

## Three tokenizers and two downloaded checkpoints carry the modality work

The multimodal machinery is delegated to components you fetch separately. The SpeechTokenizer tokenizes and reconstructs speech. Soundstorm is responsible for completing paralinguistic information, which is how prosody and the non-lexical part of a voice survive the round trip. The SEED tokenizer is used for tokenizing images. Two more checkpoints arrive on their own: the unCLIP SD-UNet weights, used to reconstruct the image, and Encodec-32k, used to tokenize and reconstruct music. The consequence is that a working AnyGPT setup pulls in four separate model families from Hugging Face, and the ones you must download yourself are fnlp/AnyGPT-base, fnlp/AnyGPT-chat, fnlp/AnyGPT-speech-modules, and the SEED tokenizer from AILab-CVC/seed-tokenizer-2. The README is explicit that the SpeechTokenizer and Soundstorm weights live in the speech-modules repository, which is a separate checkpoint download from the base and chat weights.

## Running the base model from a conda environment

The installation recipe in the README is short and specific: clone the repository, create a conda environment on Python 3.9, activate it, and install the pinned requirements. Run these four commands and you should end up in a shell where torch 2.1.0, transformers 4.34.1 and the rest of requirements.txt are installed.

```bash
git clone https://github.com/OpenMOSS/AnyGPT.git
cd AnyGPT
conda create --name AnyGPT python=3.9
conda activate AnyGPT
pip install -r requirements.txt
```

The pins matter. The requirements file asks for torch 2.1.0, torchvision 0.16.0, torchaudio 2.1.0, diffusers 0.20.2 and transformers 4.34.1, with xformers at 0.0.20 or later, and it also brings in nemo_text_processing, soundfile, librosa, hydra-core, omegaconf and flask plus gradio for the interface. Because the pins are old relative to current CUDA stacks, expect to install a matching torch build for your GPU rather than whatever pip resolves to. The repository root also holds a scripts directory, a config directory, a data directory and static assets for the demos.

## A pipe-delimited instruction format drives every base model task

The base model is driven by a single instruction format with pipes between the source modality and the target modality, and the README documents each variant with an example. The same shape covers image captioning, speech recognition, speech synthesis and music work.

```bash
text|image|A bustling medieval market scene with vendors selling exotic goods under colorful tents
image|text|static/infer/image/cat.jpg
speech|text|AnyGPT/static/infer/speech/voice_prompt2.wav
text|music|features an indie rock sound with distinct elements that evoke a dreamy, soothing atmosphere
```

The tasks the README lists are text-to-image, image caption, automatic speech recognition, zero-shot text-to-speech, text-to-music and music captioning. Zero-shot voice cloning is a fourth field, a voice prompt path appended to the text, which routes the request through the speech path with a reference clip. Speech and music responses are saved as `.wav` files and image responses as `.jpg`, with filenames made from the prompt and the time, and the paths are reported back in the response.

## Decoding strategy is a per-modality config file you can edit

Different tasks use different decoding strategies, and those strategies live in plain JSON files rather than inside the Python. The README puts image, speech and music generation in `config/image_generate_config.json`, `config/speech_generate_config.json` and `config/music_generate_config.json`, and everything that turns another modality into text in `config/text_generate_config.json`. You can modify or add parameters there to change the decoding strategy. This is the practical lever when output quality is poor, and it is more granular than the single knob most multimodal chat interfaces expose. The base model CLI itself is a script under `anygpt/src/infer/` and takes the model path, the image tokenizer path, the speech tokenizer path and config, the soundstorm path and an output directory, all as command line arguments.

```bash
python anygpt/src/infer/cli_infer_base_model.py \
--model-name-or-path models/anygpt/base \
--image-tokenizer-path models/seed-tokenizer-2/seed_quantizer.pt \
--speech-tokenizer-path models/speechtokenizer/ckpt.dev \
--speech-tokenizer-config models/speechtokenizer/config.json \
--soundstorm-path models/soundstorm/speechtokenizer_soundstorm_mls.pt \
--output-dir "infer_output/base"
```

The README also notes that generation may still be unstable because of limitations in data and training resources, and suggests generating multiple times or trying different decoding strategies.

## Why this is a research reference rather than a service dependency

Two facts in the repository limit who should build on this. The first is the stated instability of the generation, which the README attributes to data and training resource limits and answers with retries rather than with a fix. If your application needs one deterministic output, the architecture does not help you there. The second is licensing: the licence field on the repository names no standard licence, so there is no stated permission to use, modify or redistribute the code or the weights, and a reader should settle that question before anything else. Maintenance is not the concern here. The last push to the repository was on 2026-09-06, and the repository has no GitHub releases, so there is no version history to track and no tagged checkpoint matching the code. Set against that, the project page carries the demos, and the paper is linked from the README for the method itself.

## Conclusion

Use AnyGPT if you are studying unified multimodal modelling or need a working any-to-any inference path across four modalities, because the base model, chat model, tokenizers and the AnyInstruct dataset are all published and the CLI is short enough to run. Do not pick it for a product that needs stable outputs or a clear licence position, since the README states the generation may be unstable and the licence field on the repository names no standard licence. Verify first that the Python 3.9 conda environment resolves against requirements.txt, that you can reach the Hugging Face weights for fnlp/AnyGPT-base and fnlp/AnyGPT-chat, and which decoding config in the config directory matches the modality you intend to test.

## FAQ

### How do I install AnyGPT and run inference?

Clone the repository, create a conda environment on Python 3.9, activate it and install requirements.txt, then run the base model script under anygpt/src/infer/ with the model, tokenizer and output paths. The README documents the exact command lines for both the base model and the chat model.

### Which model weights do I need to download for AnyGPT?

The base weights are in fnlp/AnyGPT-base, the chat weights in fnlp/AnyGPT-chat, the SpeechTokenizer and Soundstorm weights in fnlp/AnyGPT-speech-modules, and the SEED image tokenizer in AILab-CVC/seed-tokenizer-2. The unCLIP SD-UNet weights and Encodec-32k are downloaded automatically.

### What does the AnyInstruct dataset contain?

AnyInstruct was constructed from various generative models and contains instructions for arbitrary modal interconversion. The chat model was trained on it, which is what allows free multimodal conversation with data inserted at will.

### Can I control how AnyGPT generates images, speech or music?

Yes. The decoding configuration files for image, speech and music generation sit in the config directory as image_generate_config.json, speech_generate_config.json and music_generate_config.json, and other modalities to text use text_generate_config.json. The README says you can modify or add parameters there to change the decoding strategy.

### What licence does AnyGPT use?

The licence field on the repository names no standard licence, so the README states no permission terms for the code or the weights. The repository also has no GitHub releases, though the last push was on 2026-09-06.

## Sources

- [Issues](https://github.com/OpenMOSS/AnyGPT/issues)
- [OpenMOSS/AnyGPT on GitHub](https://github.com/OpenMOSS/AnyGPT)
- [Project website](https://junzhan2000.github.io/AnyGPT.github.io/)
- [README](https://github.com/OpenMOSS/AnyGPT/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openmoss-anygpt