whisper-diarization: speaker labels for Whisper transcripts without pyannote

The gap whisper-diarization fills: transcripts without names attached

Whisper transcribes speech well and tells you nothing about who was talking. A meeting recording comes out as one undifferentiated block of text. The usual answer is pyannote.audio, a separate diarization toolkit that you then have to reconcile against Whisper's word timestamps yourself, which is where most homegrown pipelines break: the two systems disagree about where sentences start, and the speaker labels land on the wrong words.

This repository targets that reconciliation problem specifically. It is for engineers who already run Whisper locally or in a notebook and want speaker labels in the same output, without stitching two model families together by hand. The README frames it plainly: the project combines Whisper ASR with Voice Activity Detection and speaker embeddings to identify the speaker for each sentence. It is not a hosted service, it is not a Python package you import, and it does not have a web interface. It is a script and a notebook.

From raw audio to speaker-labelled text: the actual pipeline

The data flow in the README is a fixed sequence, and the order matters. First, vocals are extracted from the audio with Demucs source separation, which the project says increases speaker embedding accuracy. That is a deliberate trade: you pay a separation pass over the whole file before any transcription happens, and the README exposes --no-stem to skip it.

Whisper then produces the transcription. The timestamps are corrected and aligned with ctc-forced-aligner, which the README describes as minimizing diarization error due to time shift. This is the step that most naive Whisper-plus-diarizer scripts skip, and it is the reason the two halves agree on word boundaries at all.

Separately, the audio goes into MarbleNet for VAD and segmentation, which excludes silences, and TitaNet extracts speaker embeddings to identify the speaker for each segment. Those segments are then associated with the ctc-forced-aligner timestamps to detect the speaker for each word, and the result is realigned using punctuation models to compensate for minor time shifts. So there are two parallel tracks, acoustic and textual, joined at the word level. Whisper and NeMo parameters are coded into diarize.py and helpers.py, and the README states that CLI arguments to change them will be added later. That is the single most important constraint for anyone planning to tune this.

Installing whisper-diarization and running a first file

Python 3.10 or newer is required; the README notes 3.9 will work if you install the requirements one by one. FFMPEG and Cython are prerequisites. Install Cython first, either through pip or your distribution's package manager.

pip install cython

FFMPEG comes from your platform's package manager. The README lists Ubuntu, Arch, macOS via Homebrew, and Windows via Chocolatey, Scoop or WinGet.

# on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg

Then install the Python dependencies. The -c constraints.txt flag is in the README's command, so keep it; it pins the dependency set the project expects.

pip install -c constraints.txt -r requirements.txt

Run the pipeline against an audio file with the -a flag.

python diarize.py -a AUDIO_FILE_NAME

The output is a transcription with speaker labels per sentence. If you have at least 10GB of VRAM, the README points to diarize_parallel.py, which runs NeMo in parallel with Whisper. The README calls this experimental and says to expect errors and sharp edges, so treat it as an optimization, not the default path. Useful flags from the documented list: --no-stem disables source separation, --whisper-model defaults to medium.en, --device defaults to cuda when available, --language forces a language when detection fails, and --batch-size controls batched inference (set it to 0 for non-batched inference, and lower it if you run out of memory). --suppress_numerals transcribes numbers as pronounced letters, which the README says improves alignment accuracy.

Where whisper-diarization breaks: overlap, tuning, and the missing API

Overlapping speakers are the headline limitation, and the README says so directly: they are yet to be addressed. The suggested direction is separating the audio and isolating one speaker before feeding it into the pipeline, which the README itself notes will need much more computation. In practice this means panel discussions, heated meetings, and any recording where people talk over each other will produce labels that drift or collapse onto one speaker. If your audio is a clean interview or a lecture with one voice at a time, the design is well matched. If it is a roundtable, it is the wrong tool.

The second limitation is tuning. Because Whisper and NeMo parameters are hardcoded in diarize.py and helpers.py, changing the VAD threshold or the embedding model means editing source files, and those edits will conflict when you pull upstream changes. The README acknowledges this and promises CLI arguments later. Until then, plan on vendoring a fork or pinning a commit.

The third is packaging. There are no releases retrieved, so there is no version to pin in a requirements file. The repository layout shows a tests/ directory and a GitHub Actions workflow referenced from the README badge, so there is some test infrastructure, but the README does not document what the tests cover. There is also no documented Python API; the entry points are the two scripts and the Colab notebook. If you wanted to call this from a service, you would be shelling out to diarize.py and parsing its output, which the README does not describe in detail.

whisper-diarization vs pyannote and WhisperX: different assembly, different failure points

The search question people actually ask is whisper diarization vs whisperx, and the difference is architectural. WhisperX pairs faster-whisper with pyannote.audio for diarization and uses its own alignment model to place word timestamps. whisper-diarization takes the other route: NeMo supplies the VAD (MarbleNet) and the speaker embeddings (TitaNet), and ctc-forced-aligner does the alignment. The practical consequence is the dependency tree. This project pulls nemo_toolkit[asr] plus a forked Demucs, a forked ctc-forced-aligner, and deepmultilingualpunctuation, all from git URLs in requirements.txt. pyannote-based stacks pull pyannote.audio and typically need an accepted user agreement and an access token for the pretrained diarization models. Neither is simpler; they fail in different places. A NeMo install is heavy and can be brittle across CUDA versions, while a pyannote install depends on model access and gated downloads.

On output, the README describes punctuation-model realignment as part of this pipeline, which pyannote-based solutions generally leave to you. On overlap, pyannote's diarization is built around handling overlapping speech regions, while this project states overlap is not addressed. So the trade is: fewer moving model families and built-in punctuation handling here, versus overlap support and a token-gated but more standard stack there. Choose based on whether your recordings contain crosstalk.

Licence, upgrades and what maintenance looks like

The repository is BSD-2-Clause, a permissive licence that allows use and modification with the copyright notice retained; it does not carry the patent or trademark clauses of Apache-2.0, and it says nothing about the licences of the models the pipeline downloads. Whisper, NeMo, Demucs and the punctuation model each ship under their own terms, and the README does not enumerate them. If you are shipping a product, check those separately rather than assuming the BSD-2-Clause header covers the whole stack. This is not legal advice.

On maintenance: the last push was on 2026-08-15, and the repository is not archived. That is recent enough that the code is being touched, but there are no releases retrieved, so upgrades mean tracking the default branch. Because requirements.txt points at git URLs for Demucs, ctc-forced-aligner and the punctuation model, a fresh pip install can pull different code than it did last month even if you change nothing. Combined with hardcoded parameters in diarize.py, the realistic upgrade path is to fork, pin the git dependencies to specific commits, and rebase deliberately. The README's Future Improvements section lists only one item, a maximum sentence length for SRT output, which tells you the maintainer's stated roadmap is narrow.

Editorial conclusion

Adopt whisper-diarization if you have a GPU, Python 3.10 or newer, FFMPEG, and a batch of single-channel recordings where one person talks at a time; the pipeline runs from a single command and the parameters live in diarize.py rather than behind a CLI. Do not adopt it if you need overlapping-speaker separation, a stable Python API to import, or a published release to pin, because the README lists overlapping speakers as unaddressed and there are no releases retrieved. Before committing, check the VRAM headroom for your chosen --whisper-model, confirm your audio is mono and clean enough for Demucs separation, and read diarize.py to see which Whisper and NeMo parameters you would have to edit by hand.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-2-Clause
3. MahmoudAshraf97/whisper-diarization on GitHub
4. README