# MOSS-Audio: one audio model for speech, sound and music

> MOSS-Audio is an open source audio understanding model that handles transcription, speaker and emotion analysis, environmental sound, music, captioning and time-aware question answering in a single architecture, released in 4B and 8B Instruct and Thinking variants. It suits research and evaluation work on audio models, and it is a heavy download that asks for a GPU and a trained reader of the technical report.

**OpenMOSS/MOSS-Audio** — An open-source model for understanding speech, environmental sounds, and music through captioning, question answering, and reasoning

- Repository: https://github.com/OpenMOSS/MOSS-Audio
- Website: https://openmoss.ai/MOSS-Audio/
- Stars: 691 · Forks: 45
- Language: Python
- License: not declared
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/openmoss-moss-audio

## One model for the things a transcription model cannot do

MOSS-Audio is an open source audio understanding model from MOSI.AI, the OpenMOSS team and the Shanghai Innovation Institute, and the project's own framing is that understanding audio requires more than transcribing words. It demands perceiving acoustic cues, recognising speakers and emotions, interpreting environmental sounds, reasoning over temporal context and handling complex multi-step inference, and the model is built to unify those capabilities in a single model.

The capabilities are listed as seven groups. Speech and content understanding covers recognition and transcription with both word-level and sentence-level timestamp alignment. Speaker, emotion and event analysis identifies speaker characteristics, analyses emotional state from tone, timbre and context, and detects key acoustic events. Scene and sound cue extraction pulls cues out of background sound, environmental noise, music and non-speech signals to infer context and atmosphere. Music understanding analyses style, emotional progression, instrumentation and salient acoustic features. Audio question answering and summarization covers speech, podcasts, meetings, interviews and environmental recordings. Time-aware QA handles questions that reference a moment in the audio. Complex reasoning does multi-hop reasoning over audio content, powered by chain-of-thought training and reinforcement learning.

The choice to unify rather than specialise is the interesting architectural bet. Most audio tooling today is a stack: a speech-to-text model, a separate speaker diarization tool, an audio tagging model, a music classifier. A single model that does all of it removes the orchestration, and the price is that one model has to be competent at all of it rather than excellent at one thing.

The repository is Python, with app.py, infer.py, a src/ package, a demo/ directory containing a small web app with its own package.json, a finetune/ directory and a test/ directory. There is a separate moss_audio_usage_guide.md document and a bilingual readme, with README_zh.md for Simplified Chinese.

## Four checkpoints, and what Thinking actually changes

The release provides four models: MOSS-Audio-4B-Instruct, MOSS-Audio-4B-Thinking, MOSS-Audio-8B-Instruct and MOSS-Audio-8B-Thinking. The distinction the documentation draws is between the Instruct variants, optimized for direct instruction following, and the Thinking variants, which provide stronger chain-of-thought reasoning capabilities.

So the two axes are parameter count and reasoning style, and they are orthogonal. A team that wants a captioner that answers immediately takes an Instruct model, and a team that wants the model to show its work on a hard question takes a Thinking model. Whether the extra capability is worth the extra tokens and latency is a question the documentation does not attempt to answer, and the evaluation section is where you would look for it.

The weights are distributed through two model hubs, a Hugging Face collection under the OpenMOSS-Team organisation and a ModelScope collection, and there is a project page at openmoss.ai/MOSS-Audio/. A technical report sits on arXiv at 2606.01802, released on 2026.6.1 according to the news section.

The news list also gives a useful timeline. The model itself was released on 2026-04-13, and the fine-tuning code and documentation were added on 2026-04-20, with the LoRA and full-parameter training examples in finetune/FINETUNE.md. The repository publishes no GitHub releases, so the versions you install come from the model hubs rather than from tags, and the last push to the repository was on 2026-09-06.

## Encoder, adapter, language model, and a frame rate

The architecture is modular, with three components: an audio encoder, a modality adapter and a large language model. Raw audio is encoded by MOSS-Audio-Encoder into continuous temporal representations at 12.5 Hz, which are projected into the language model's embedding space through the adapter and then consumed by the LLM for autoregressive text generation.

The 12.5 Hz figure is the number to hold on to. It is the rate at which the encoder produces representations, so it is the model's effective temporal resolution for anything that depends on when something happened. For timestamp ASR and event localisation to work at all, that rate has to be high enough, and 12.5 frames per second means roughly 80 milliseconds per frame, which is coarse for phoneme-level timing and fine for locating a car door or a change of speaker.

The other architectural decision is that the audio frontend is not off the shelf. The documentation says a dedicated encoder is trained from scratch rather than relying on existing audio frontends, in order to get speech representations that hold up better, tighter temporal alignment and better extensibility across acoustic domains. Training your own encoder costs a great deal more than fine-tuning a pretrained one, and the payoff is a model that is not tied to the assumptions of an existing frontend, including its channel assumptions and its training data mix.

The adapter is where the audio world meets the language world, and it is the component that lets an LLM consume audio at all: the encoder's output is a sequence of vectors in its own space, and the adapter projects them into the embedding space the language model already understands.

## DeepStack injection, and time markers in the pretraining data

Two mechanisms sit between the encoder and the language model, and both are there to solve a specific problem with audio.

The first is DeepStack cross-layer feature injection. The problem it addresses is that using only the encoder's top-layer features tends to lose low-level prosody, transient events and local time-frequency structure. The solution is to select features from earlier and intermediate layers as well as the final layer, project them independently, and inject them into the language model's early layers, preserving information across the range from low-level acoustic detail to high-level semantic abstraction. The documentation argues this is well suited to audio, because it retains rhythm, timbre, transients and background structure, which a single high-level representation cannot fully capture.

That is a sensible argument. A top-layer representation of audio has already discarded much of what makes an event an event, and injecting intermediate features gives the language model a chance to use them. The cost is that more features have to be projected and pushed through the LLM's early layers, which is more compute and more parameters than a single-stream design.

The second mechanism is time-aware representation through time-marker insertion. During pretraining, explicit time tokens are inserted between audio frame representations at fixed intervals to indicate temporal positions. The stated effect is that the model learns what happened when within a unified text generation framework, which naturally supports timestamp ASR, event localisation, time-based QA and long-audio processing.

Inserting tokens during pretraining rather than deriving time at inference is the interesting choice. It means the temporal information is part of the sequence the model was trained on, so a question like what happened at 2 minutes 15 seconds is answerable in the same text generation format as everything else, with no separate alignment head.

## Serving, fine-tuning and the packaging around the model

The repository is a working package rather than a bare model card, and the pyproject.toml shows what it expects.

The base dependencies are modest and mostly about loading weights and audio: safetensors, numpy, soundfile, tiktoken, einops, scipy, tqdm, packaging, gradio, requests and streamlit. Torch is not among them, which is deliberate. It sits in an optional extra called torch-runtime together with accelerate, a pinned torch and torchaudio for Linux and macOS, torchcodec, and a pinned transformers. The Python requirement is 3.10 or newer, and the package version is 0.1.0, named moss-audio and built with setuptools.

There is a second optional extra for flash attention, with platform-conditional markers so that the flash-attn package installs on Linux and mps-flash-attn on macOS. Getting the attention implementation right is a real part of running an 8B audio model at usable speed, and having the platform split handled in the metadata rather than in a wiki page is the right call.

The quickstart sections in the readme cover environment setup, basic usage, fine-tuning, a Gradio app and SGLang serving, and the repository layout matches: app.py is the Gradio application, infer.py is the inference entry point, the demo/ directory holds a small web front end with its own JavaScript and styles, and finetune/ holds the training code.

The fine-tuning documentation is explicitly about LoRA and full-parameter training, which is the practical part of adopting a 4B or 8B model. The evaluation section is a separate heading in the contents, and for a project that publishes no GitHub releases and no benchmark numbers in the visible readme, that section and the technical report are where you would look before drawing conclusions.

## Where MOSS-Audio is the wrong choice

The first limitation is size and the hardware it assumes. An 8B parameter multimodal model with a dedicated audio encoder is not something you run on a laptop CPU, and the optional torch-runtime extra pins specific CUDA and macOS builds rather than letting you pick. For a team whose audio work is a 200-call-a-day transcription service, this is a research model used at research scale, and the fine-tuning code for LoRA and full-parameter training exists because adapting it is the expected workflow, not an exception.

The second is the licence question, and the repository is not decisive about it. The project metadata records the licence as unknown, while the package metadata points at a LICENSE file, and the readme itself does not summarise the terms. For a model released through the OpenMOSS team on Hugging Face and ModelScope, the terms that matter for commercial use are the ones on the model hub pages rather than the ones in this repository, and the readme does not point you at them explicitly. Read both before putting the weights behind a product.

The third is that unified is a bet on generality, not a guarantee of quality. The model claims to handle speech, environmental sound and music with one architecture, and a single model that is good at seven things is rarely as good at one of them as a specialist. If your requirement is word-perfect transcription for subtitles, a dedicated ASR model with better timestamp behaviour is the safer dependency, and the unified model is the better choice when you need a single system that can answer a question about a sound that is not speech at all.

The fourth is evaluation. The visible documentation points to an evaluation section and a technical report, and it publishes no scores in the readme itself, so a team choosing between this model and a hosted API has to do that comparison work rather than reading a table. That is normal for a model release and it is still a cost.

The fifth is operational maturity. There are no GitHub releases, the package version is 0.1.0, the last push was on 2026-09-06, and the deployment paths in the readme are a Gradio app, a Streamlit dependency and SGLang serving. That is a demo-and-research shape rather than a production service shape, and the difference matters if you are planning to run this on incoming calls.

## Conclusion

MOSS-Audio fits a research team that needs one model covering speech, environmental sound and music rather than a pipeline of separate ones, and that can afford 4B or 8B weights on a GPU. It is the wrong choice for a production voice agent that needs predictable latency, for a team that wants to fine-tune on proprietary audio without a research budget, and for anyone who needs a licence it can rely on, because the project metadata records the licence as unknown even though the package metadata points at a LICENSE file. Before adopting it, read the technical report on arXiv, check the model weights and their licences on the Hugging Face and ModelScope collections, and note that the last push was on 2026-09-06, with the model released on 2026-04-13 and the fine-tuning code added on 2026-04-20.

## FAQ

### What does MOSS-Audio do?

It is an open source audio understanding model that performs unified modeling over real-world audio, covering speech understanding, environmental sound understanding, music understanding, audio captioning, time-aware question answering and complex reasoning, in a single model rather than a pipeline of separate tools.

### Which MOSS-Audio models are available?

Four: MOSS-Audio-4B-Instruct, MOSS-Audio-4B-Thinking, MOSS-Audio-8B-Instruct and MOSS-Audio-8B-Thinking. The Instruct variants are optimized for direct instruction following and the Thinking variants for chain-of-thought reasoning.

### What is the architecture of MOSS-Audio?

It is modular, with an audio encoder, a modality adapter and a large language model. The MOSS-Audio-Encoder turns raw audio into continuous temporal representations at 12.5 Hz, the adapter projects them into the language model's embedding space, and the LLM consumes them for autoregressive text generation.

### How do I fine-tune MOSS-Audio?

The fine-tuning code and documentation were added on 2026-04-20, with LoRA and full-parameter training examples in finetune/FINETUNE.md. The torch runtime, including pinned torch, torchaudio, torchcodec and transformers versions, is an optional extra named torch-runtime.

### Where can I get the MOSS-Audio model weights?

From the Hugging Face collection under the OpenMOSS-Team organisation, or from the ModelScope collection, with a project page at openmoss.ai/MOSS-Audio/. The technical report is on arXiv as 2606.01802, and the repository was last pushed to on 2026-09-06.

## Sources

- [Issues](https://github.com/OpenMOSS/MOSS-Audio/issues)
- [OpenMOSS/MOSS-Audio on GitHub](https://github.com/OpenMOSS/MOSS-Audio)
- [Project website](https://openmoss.ai/MOSS-Audio/)
- [README](https://github.com/OpenMOSS/MOSS-Audio/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openmoss-moss-audio