WhisperLiveKit: self-hosted streaming ASR with diarization and an OpenAI-compatible API

What WhisperLiveKit is solving that plain Whisper does not

Whisper expects a complete utterance. Feed it two-second chunks and it loses the context that disambiguates homophones, and it can cut a word in half at the chunk boundary. WhisperLiveKit exists to avoid that failure mode. The README frames the problem directly: "Why not just run a simple Whisper model on every audio batch? Whisper is designed for complete utterances, not real-time chunks." The project's answer is a buffering and commit policy borrowed from simultaneous translation research, so the transcript is emitted incrementally but only when the model is confident enough to commit those words.

The target user is an engineer building a live captioning, meeting-notes or call-analytics feature who does not want to send audio to a third-party API. Everything runs locally: the ASR models, the voice activity detector, the diarization model and the translation model. The server is FastAPI-based and supports multiple concurrent users, with voice activity detection reducing compute when nobody is speaking. If your requirement is "transcribe a folder of recordings overnight", this is more machinery than you need.

The streaming policy: AlignAtt, LocalAgreement and causal encoders

The core mechanism is a commit policy. Simul-Whisper and SimulStreaming use the AlignAtt policy, which inspects cross-attention to decide when a token is stable enough to emit. WhisperStreaming uses the older LocalAgreement policy instead. Either way, the server holds a growing audio buffer, runs the model repeatedly over that buffer, and emits only the prefix that two consecutive hypotheses agree on. The user-visible effect is a transcript that appends rather than rewrites, at the cost of some latency between speech and text.

Diarization runs alongside transcription. The README points to Streaming Sortformer, which assigns speaker labels in real time rather than in a separate post-processing pass. That matters architecturally: you cannot run diarization as a batch step afterwards if you want speaker labels to appear live, so the server carries a second model in memory. The compose.yml file shows the memory cost, since the GPU diarization service builds with the diarization-sortformer extra and runs with --diarization.

The translation path is separate again. It uses NLLW, built on a distilled NLLB-200 model via CTranslate2, and the README claims simultaneous translation from and to 200 languages. Translation is an optional extra, not part of the base install, and the server only accepts a target_language query parameter when it was started with --target-language. The Qwen3-ASR-causal entry in the README describes a different trade-off: a causal encoder that processes each audio block exactly once, giving constant compute per audio second and append-only transcripts, instead of re-running the encoder over a growing buffer.

Installing WhisperLiveKit and running a first transcription

The package is on PyPI as whisperlivekit and requires Python 3.11 through 3.13. The README's install line is a single pip command. The base install pulls torch, torchaudio, faster-whisper, FastAPI, uvicorn and librosa, so expect a large download.

pip install whisperlivekit

Once installed, the wlk command starts the server. The README's quick start uses the base model and English, and tells you to open http://localhost:8000 in a browser and start talking.

wlk --model base --language en

The same CLI can pull models ahead of time and manage what is on disk, which is worth doing before a demo so the first request does not stall on a Hugging Face download.

wlk models
wlk pull large-v3

For a file rather than a live microphone, the transcribe subcommand runs without starting a server. The README shows both a plain transcription and subtitle generation to SRT.

wlk transcribe meeting.wav
wlk transcribe --format srt podcast.mp3 -o podcast.srt

If you would rather not manage the Python environment, the repository ships a compose.yml with three services: a GPU service with Sortformer diarization on port 8000, a GPU Voxtral service on port 8001, and a CPU service on port 8000. They share a named volume for the Hugging Face cache, and the GPU services declare gpus: all. The Dockerfile installs CUDA 12.9 runtime images and uses uv to sync extras, so the build context needs submodules initialized when the qwen3 extras are used.

Talking to the server: WebSocket parameters and the OpenAI-compatible route

The native streaming endpoint is ws://localhost:8000/asr. Per-session behaviour is controlled through query parameters: language sets the transcription language for that session, target_language sets the translation target, and context passes terminology or a phrase list to condition the session. The README notes that context is supported by Whisper-family and SimulStreaming backends, so it is not universal across backends. That is a real constraint if you plan to switch backends later.

There is also a mode parameter. Its default is full, which resends the entire transcript state on each update; mode=diff switches to an incremental snapshot and diff protocol. The README labels diff as experimental and aimed at integrators building their own client, and notes that the bundled web UI uses full. If you write a custom client, the diff protocol means less data over the wire but more client-side state handling, and the experimental label means the wire format is the part most likely to move between releases.

For non-streaming use, the server exposes an OpenAI-compatible REST subset. The README's example is a curl call to /v1/audio/transcriptions with a file field, and the same base URL works with the OpenAI Python SDK with an api_key of "unused". The README is explicit that these are compatibility-oriented subsets, and points to docs/API.md for the supported options. Treat the word subset as load-bearing: if your integration depends on an OpenAI audio parameter that is not listed there, assume it is not implemented. A Deepgram-compatible WebSocket also exists, with the same caveat about supported options.

Where WhisperLiveKit gets in the way

The dependency surface is the first obstacle. The project pins torch>=2.6.0 and torchaudio>=2.0.0, and the optional extras are split by hardware: cu129 for CUDA 12.9, cpu for a CPU PyTorch stack, mlx-whisper and voxtral-mlx for Apple Silicon, qwen3-vllm for CUDA, and separate extras for translation, sentence tokenization, FunASR and Voxtral. Picking the wrong combination means either a CPU-only install that is too slow for streaming or a CUDA build that does not match the driver on the host. The README links a troubleshooting guide for GPU setup and environment issues, which is a fair signal that this is where users get stuck.

Streaming also costs you accuracy relative to batch. The whole point of the commit policy is to emit text before the utterance is complete, and the README's own framing is that Whisper is designed for complete utterances. A batch transcription of the same audio, run after the fact, has more context available. If your workload tolerates a delay, batch wins on quality.

Finally, this is a beta-classified project. The pyproject.toml classifier is Development Status :: 4 - Beta, and the last release in the repository is v0.2.26 from 2026-08-29. The repository's most recent push was on 2026-09-14. The README does not document a rollback procedure or a compatibility policy for the WebSocket protocol between versions, so pinning a version in production is a decision you have to make yourself.

WhisperLiveKit versus running faster-whisper yourself

The closest alternative is to build on faster-whisper directly, which WhisperLiveKit itself depends on. The difference is what each one owns. faster-whisper gives you a model and a transcription call; you own the audio capture, the buffering strategy, the WebSocket server, the session management and the commit logic. WhisperLiveKit owns all of that and adds diarization and translation as optional layers.

That makes the choice a question of whether the streaming policy is your problem or theirs. If you are building a live captioning feature and you do not want to implement AlignAtt or LocalAgreement yourself, WhisperLiveKit is the shorter path. If you are transcribing uploaded files, or if you need a custom commit policy tuned to your domain, faster-whisper is the smaller dependency and leaves the architecture to you.

A second comparison is with hosted APIs. The README notes OpenAI and Deepgram compatibility, which is aimed at teams migrating off a hosted service. The trade here is operational: you stop paying per minute and stop sending audio off-premises, and in exchange you run a GPU, pull multi-gigabyte model weights, and take on the upgrade treadmill yourself.

Licence and the cost of staying current

WhisperLiveKit is Apache-2.0, and the pyproject.toml declares license = "Apache-2.0" with license-files = ["LICENSE"]. That is a permissive licence, but it covers this project's code, not the model weights it downloads. Whisper-family models, the NLLB-based translation model, Sortformer and Qwen3-ASR weights each carry their own terms, and the repository does not consolidate them. If you are shipping a product, check the licence of every model you pull with wlk pull, not just the repository licence. This is not legal advice.

Upgrade cost is mostly model and dependency churn. Releases in the repository land roughly monthly: v0.2.24 on 2026-07-11, v0.2.25 on 2026-08-01, v0.2.26 on 2026-08-29. The project tracks fast-moving research, and the README cites papers from 2025 and 2026, so a pinned version will drift away from upstream reasonably quickly. The uv.lock file means Docker builds are reproducible if you build from a fixed commit, which is the cheapest way to make upgrades deliberate rather than accidental.

Editorial conclusion

Adopt WhisperLiveKit if you need a self-hosted streaming transcription server with diarization, translation and OpenAI-compatible endpoints, and you have a machine with a supported GPU or Apple Silicon. Do not adopt it if you only need to transcribe uploaded files on a schedule, since a batch tool such as faster-whisper is simpler and has no streaming machinery to configure. Before committing, verify three things on your own hardware: that your CUDA version matches one of the cu129 or cpu extras, that the model you intend to use is pulled with wlk pull, and that your client speaks the WebSocket protocol at ws://localhost:8000/asr or the REST route at /v1/audio/transcriptions.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. QuentinFuxa/WhisperLiveKit on GitHub
4. README
5. Releases