DiariZen turns a self-supervised speech model into a speaker diarization pipeline

The pipeline is a Pyannote-shaped API over a pruned WavLM

DiariZen is a toolkit for speaker diarization, driven by AudioZen and by Pyannote 3.1. The project listing calls it a speaker diarization toolkit, and the package metadata names the two upstreams directly: AudioZen is mainly maintained by Xiang Hao, and DiariZen itself is mainly maintained by Jiangyu Han.

The inference path is the familiar pyannote shape, which is what makes the toolkit approachable. You load a pipeline, point it at a wav file, and iterate over the result:

from diarizen.pipelines.inference import DiariZenPipeline

# load pre-trained model
diar_pipeline = DiariZenPipeline.from_pretrained("BUT-FIT/diarizen-wavlm-large-s80-md")

# apply diarization pipeline
diar_results = diar_pipeline('./example/EN2002a_30s.wav')

# print results
for turn, _, speaker in diar_results.itertracks(yield_label=True):
    print(f"start={turn.start:.1f}s stop={turn.end:.1f}s speaker_{speaker}")

The result object exposes itertracks with yield_label, and each turn carries a start, an end and a speaker label, so the output is the same tuple of turn information a pyannote pipeline gives you.

There is also an RTTM path, which is the standard evaluation format for diarization systems. You pass a session name and an output directory, and the pipeline writes the RTTM result for that session while still returning the in-memory result:

# load pre-trained model and save RTTM result
diar_pipeline = DiariZenPipeline.from_pretrained(
        "BUT-FIT/diarizen-wavlm-large-s80-md",
        rttm_out_dir='.'
)

The pre-trained checkpoints are on Hugging Face, including the base and large s80 models and a v2 of the large model.

Structured pruning is the contribution, not the diarization wrapper

The benchmark section explains the approach in a sentence: the models are trained on a compound dataset and then structured pruning removes redundant parameters. The research framing in the citations makes it clearer. The three papers are about leveraging self-supervised learning for speaker diarization, about fine-tuning before structured pruning towards compact and accurate self-supervised models, and about efficient and generalizable diarization via structured pruning of self-supervised models.

The order matters. Fine-tune first, then prune, is the argument of the second paper, and it is the reason the checkpoints carry an s80 marker in their names. Pruning a self-supervised model before adapting it to a downstream task would damage the representations the adaptation depends on.

The benchmark table is a Diarization Error Rate comparison across eight datasets, with Pyannote v3.1 and three DiariZen checkpoints, base s80, large s80 and large s80 v2. The README is careful about the conditions, and those conditions are what make the numbers meaningful. AISHELL-4 was converted to mono using sox, NOTSOFAR-1 contains only single-channel recordings, DER is evaluated without applying a collar, no domain adaptation is applied to any individual dataset, and all experiments use the same clustering hyperparameters across datasets.

The results are consistent rather than spectacular. On AMI-SDM the table reads 22.4 for Pyannote v3.1 against 15.8, 14.0 and 13.9 for the DiariZen checkpoints. On VoxConverse it is 11.3 against 9.7, 9.2 and 9.1. The pattern is that the self-supervised base beats the reference system on every dataset listed, and the gap narrows as the DiariZen variants get larger, with the v2 checkpoint taking the best score on AliMeeting far at 10.8 and on NOTSOFAR-1 at 16.7.

Installing it is a conda environment, a pinned torch and two submodules

The installation section is a sequence of shell steps, and the version pinning is not optional decoration.

# create virtual python environment
conda create --name diarizen python=3.10
conda activate diarizen

Then torch, torchvision and torchaudio are installed at fixed versions from the CUDA 12.1 wheel index:

# install pytorch (CUDA 12.1 build)
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 \
    --index-url https://download.pytorch.org/whl/cu121

The DiariZen package itself is installed in editable mode from its requirements file, and then pyannote-audio is installed from the vendored copy in the repository:

# install diarizen 
pip install -r requirements.txt && pip install -e .

# install pyannote-audio
cd pyannote-audio && pip install -e .[dev,testing] -c ../constraints.txt && cd ..

The last step pulls in the dscore scoring tool, which comes in as a git submodule:

# install dscore
git submodule init
git submodule update

The requirements list is long and includes the usual speech stack, librosa, soundfile, numpy pinned at 1.26.4, scipy, pandas, einops, h5py, pystoi for the objective functions, pesq for the perceptual quality metric, onnxruntime-gpu, and accelerate at 1.6.0. The dependencies also list torchinfo and thop, the latter being the tool used to count parameters, which is what a pruning project needs to report the size reduction it achieved.

The weights are non-commercial, and the reason is a licence conflict

The licence section is the most consequential part of this repository, and it splits the project in two.

The code in the repository is MIT licensed. The pre-trained model weights are released under CC BY-NC 4.0, and downloading them means agreeing to the non-commercial terms.

The reason is spelled out in the compliance note, and it is a genuine conflict rather than caution. Certain datasets in the training mixture, RAMC, MSDWild and DIHARD-3, explicitly prohibit commercial use, while others such as AISHELL-4 and AliMeeting carry a ShareAlike licence. Under the assumption that model weights could be viewed as derivative works, the must allow commercial use and must forbid commercial use requirements collide.

The resolution the project chose is to apply the NC restriction to the whole release, prioritising the protection of the non-commercial data providers and preventing unauthorized commercial exploitation. The README states that the release does not grant a commercial licence even for the portions of the model potentially influenced by ShareAlike data, that use is for research and academic purposes only, and that you assume all legal risks if you attempt to use the weights in a commercial product.

For a research group this is a reasonable arrangement, and the reason for it is sound. But it means the boundary between the MIT code and the NC weights is the boundary you have to reason about, and if you are building a product that transcribes and attributes speakers, the pre-trained checkpoints are not available to you. The code, the recipes and the training pipeline still are.

Training and pruning live in recipes, and the package is pre-alpha

The usage section is short and points at shell scripts rather than a Python API for the training side. Model training is in recipes/diar_ssl/run_stage.sh, and model pruning is in recipes/diar_ssl_pruning/run_stage.sh.

That the training entry point is a shell script rather than a module invocation tells you something about how this is operated. The project is built around a training pipeline with stages, and the scripts are the interface, so the same pattern a speech lab would use for an experiment sweep is what you get here.

The package metadata is where the maturity signal is. The version is 0.0.1, the Development Status classifier is 2 - Pre-Alpha, and the build backend is flit_core with semantic release configured to bump the version in pyproject.toml and create GitHub releases automatically, with upload_to_repository set to false so nothing is pushed to PyPI automatically.

The repository has no GitHub releases as of the listing, and the last push was on 2026-08-04 with the repository not archived. The updates section gives the recent history: multi-channel WavLM support for speaker diarization on 2026-01-31, updated benchmarks with the v2 model on 2025-12-09, and on 2025-06-03 the structured pruning recipes were uploaded along with new pre-trained models and updated benchmark results.

The primary language being listed as Jupyter Notebook is a fair description of a research repository, and it is a reminder about what you are adopting. This is a lab's working codebase with a research interface, and the pre-alpha marker plus a 0.0.1 version mean the Python API may still move. The checkpoints on Hugging Face, with the model card and the paper, are the more stable artifacts to build against.

Where a commercial diarization API is the better answer

The comparison that matters for a production team is not against another research toolkit, it is against a hosted diarization service, and the difference in approach is straightforward. A hosted API gives you a contract, a supported Python client, a latency target and a per-hour price. DiariZen gives you a model you control, running on hardware you own, that you can fine-tune on your own audio and that you can inspect.

That trade is worth taking when the audio is sensitive, when you need to adapt the model to a domain the published checkpoints do not cover, or when the per-minute cost of a hosted API at your volume is the dominant line in the budget. It is not worth taking when you want a diarization feature in a product, because then you are the one responsible for the GPU, the model updates and the accuracy of the output in every language you support.

The second limit is the licence, and it is not a soft one. A team that wants to ship speaker attribution in a commercial product cannot use the released weights, and retraining from scratch is a research project rather than an integration task. The code is available under MIT, so building your own checkpoint is permitted, but the cost of that path is exactly the cost the published models were meant to avoid.

The third limit is the evaluation gap. The benchmark table covers eight well known corpora with DER reported without a collar and with fixed clustering hyperparameters. That is the right way to compare research systems, and it is not what tells you whether the system works on your call centre recordings with two speakers who overlap. A collar in particular changes DER by a meaningful margin, so a number quoted without one is not directly comparable to a production expectation.

The alternative inside the research space is to build on the pyannote-audio pipeline directly, which the repository already vendors and installs as part of setup. The difference in approach is that DiariZen substitutes a pruned self-supervised backbone and a training recipe for the reference system, and the pruning is what the papers are about.

Editorial conclusion

Adopt DiariZen if you are doing research on speaker diarization and want a self-supervised WavLM model that has been pruned rather than one of the standard commercial APIs, and if your use is research or academic. Do not adopt the released weights for a commercial product, because they are under CC BY-NC 4.0 and the README states the release does not grant a commercial licence even for parts of the model influenced by ShareAlike data. Verify first which checkpoint you need, that your environment matches the pinned versions, and that the diarization error rate you need is met on your own audio rather than only on the published benchmark table.

Frequently asked questions

Official sources

1. BUTSpeechFIT/DiariZen on GitHub
2. Issues
3. License: MIT
4. README