MOSS-Transcribe-Diarize labels speakers in the same pass as the words

Transcribing and separating speakers in the same pass

MOSS-Transcribe-Diarize is a 0.9 billion parameter model that transcribes long recordings and labels who is speaking, jointly, in one forward pass. The README is explicit that this replaces the usual arrangement of stitching a recognition system to a separate diarization system.

That framing identifies the problem precisely. A two-stage pipeline transcribes first and assigns speakers second, which means the speaker boundaries and the word timings are produced by different models with different notions of where a segment ends. Errors compound exactly where conversations are hardest: overlapping speech, quick exchanges, an interruption mid-sentence. Doing both at once means one model resolves the boundary rather than two models disagreeing about it.

The target material is stated as meetings, calls, podcasts, interviews, lectures and video, and the phrase used is long, messy, multi-speaker recordings. The audience is anyone whose transcription problem is not one person speaking clearly into a microphone, which is most real transcription problems, and the model supports more than fifty languages.

The output format does the structural work

The model emits a compact stream rather than a document, and the format repays a moment's attention because it determines how easy the output is to consume.

Each segment carries a start time, a speaker tag, the transcribed text and an end time, with timestamps in seconds and adjacent segments concatenated into a single stream. Speaker tags are sequential labels, so the first voice encountered becomes the first tag and so on. Both timestamps sit on the segment rather than being inferred from neighbours, which means a consumer can cut audio at exactly the boundaries the model asserted.

Two properties follow that matter downstream. The labels are consistent across the whole recording rather than per window, which is the actual hard part of diarization: recognising that the voice at minute forty is the same one from minute two. And because the format is a flat stream of delimited fields rather than nested structure, parsing it into subtitles or a database row per utterance is straightforward.

Optional acoustic event annotations can be emitted alongside, so a downstream system can know that something other than speech occurred. Custom prompts and hotword support are documented as well, which is the conventional route to getting proper nouns and domain vocabulary spelled correctly.

A small model built from familiar parts

The architecture table is unusually legible, and every component is something a reader will already know.

The text backbone is a causal decoder in the style of a 0.6 billion parameter Qwen3 model. The audio encoder uses a Whisper-Medium encoder configuration, fed by the standard Whisper feature extractor at 16 kHz with 80 mel bins in 30 second chunks. Between them sits a bridge that merges audio features temporally by a factor of four and passes them through a small adaptor network. Fusion happens by substituting the audio features into the embedding sequence where audio placeholder tokens sit.

That is a well-trodden recipe, and using it is a point in the project's favour rather than against. The contribution here is the training and the output format rather than a novel architecture, and building on known components means the behaviour of each part is understood and the whole thing fits in 0.9 billion parameters.

The temporal merge is the detail with the most practical consequence. Reducing the audio sequence length fourfold before it reaches the language model is what makes long-form input affordable, since the decoder's costs grow with sequence length and an hour of audio is a great many frames.

Installing it, and the serving paths on offer

The manifest sets a reasonable interpreter floor and, notably, uses ranges rather than exact pins.

requires-python = ">=3.10"

The dependency style deserves credit. The transformers library is constrained to a major version range rather than a single release, numpy is bounded on both sides with room to move, and the audio libraries carry minimums. Research code that can coexist with a normal environment is rare enough to be worth pointing out. Torch itself is an optional dependency group rather than a hard requirement, alongside a separate group for an attention implementation, so the base install stays light and you choose the runtime.

Two console commands are installed with the package, one producing subtitles from the command line and one launching a web interface for the same job. A web application for subtitling is a sensible thing to ship with a model like this, since the people who need speaker-labelled subtitles are not all going to write Python.

For scale, the README documents serving through two separate high-throughput inference servers, and use within an established speech toolkit's ecosystem. Offering more than one serving path matters because it means the model is packaged conventionally enough that general purpose servers can load it, rather than requiring a bespoke harness.

The best model is not the open one

The first sentence of the introduction contains the limitation that should shape any evaluation: a stronger model exists, is called Pro, has higher overall performance, and is available through an online playground rather than as weights.

That is a fair arrangement and it needs to be read plainly. The open 0.9 billion parameter model is the entry point, and the ceiling of what this team can do is behind a hosted service. Anyone benchmarking the open model against commercial competitors should understand they are comparing the smaller sibling, and anyone whose accuracy requirement is not met by it faces a hosted service rather than a larger download.

Other limits are ordinary. There are no tagged releases, so pin a commit. The evaluation uses character error rate and its speaker-aware variants across four test sets, with comparisons against named commercial systems, and those numbers were produced by the authors in their own setup on benchmarks weighted toward Chinese material, which is the context to keep when reading them.

The competition result reported for July 2026 is the more independent evidence, since a challenge is judged by someone else on data the entrants did not choose. It covers fourteen languages, which is fewer than the fifty the model claims to support, so it validates part of the claim rather than all of it.

Stitching two systems together is the alternative, and the seam is the difference

The alternative is the conventional pipeline this project was built to replace: a recognition model producing a transcript, and a separate diarization system deciding who spoke when, with the two aligned afterwards.

The case for that arrangement is real. Each component can be chosen independently, so you can pair the strongest recognition model for your language with whichever speaker system suits your audio, and swap either without touching the other. If your recognition needs are unusual, or you already run a transcription service you trust, adding diarization to it is less disruptive than replacing everything.

The case against is the seam. Alignment between two models' notions of a segment is where two-stage pipelines fail, and they fail worst on the recordings people most need this for: crosstalk, interruptions, short back-and-forth exchanges. A joint model emits the boundary and the label together, so there is nothing to reconcile.

Choose the pipeline when your components are already chosen and your audio is orderly. Choose this when speaker attribution on difficult recordings is the requirement, or when running one 0.9 billion parameter model is operationally simpler than maintaining two systems and the alignment code between them.

Apache terms, fine-tuning, and what to check first

The project is Apache-2.0 licensed with the file present, which includes an express patent grant and permits commercial use. Model weights are distributed through a model hub and carry their own terms, a separate question from the licence on this code, and worth reading directly before shipping. This is not legal advice.

The repository is compact and complete for its purpose: the inference package, an example directory, a test directory, a fine-tuning script and a dedicated fine-tuning document. Shipping fine-tuning as documented rather than implied matters here, because a model whose output format is this specific is one people will want to adapt to their own speaker conventions and vocabulary. Documentation exists in English and Chinese, and the subtitle interface was extended to both languages in July 2026. The last push was on 2026-09-15.

For an evaluation, the order is straightforward. Take one of your genuinely difficult recordings, the meeting with crosstalk rather than the clean interview, and compare the speaker labels against what you know happened. Character error rate on a benchmark will not tell you whether the tags stay consistent across an hour, and that consistency is the thing this model is offering.

Editorial conclusion

MOSS-Transcribe-Diarize is the right model when speaker attribution on difficult recordings is the requirement, when you want timestamps and consistent labels from one pass rather than an alignment step you maintain, and when a 0.9 billion parameter model on your own hardware suits the deployment. It is the wrong choice when your accuracy bar is set by the strongest system available, because the README states a stronger Pro model exists only through a hosted playground. Pin a commit, since there are no tagged releases, and evaluate on a recording with genuine crosstalk rather than a clean interview, because label consistency across an hour is the claim worth testing and no character error rate will show it.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. OpenMOSS/MOSS-Transcribe-Diarize on GitHub
4. Project website
5. README