collabora/WhisperLive: a nearly-live Whisper transcription server you run yourself

What WhisperLive actually solves for a developer

Whisper is a batch model. You hand it a file, it returns a transcript. Turning that into something that reacts while a person is still talking means solving chunking, buffering, model loading and transport yourself, and that plumbing is where most hobby transcription projects stall. WhisperLive is that plumbing, packaged as a server. The README describes it as "a real-time transcription application that uses the OpenAI Whisper model to convert speech input into text output", usable for live microphone input and for pre-recorded files. The audience is developers: the package metadata lists Intended Audience as Developers and Science/Research, and the repository ships browser extensions for Chrome and Firefox, an iOS client, a Docker directory and a Python client script rather than a consumer application. If you want to bolt streaming transcription onto a meeting tool, a captioning overlay or an internal note-taking service, this is the layer you would otherwise write.

Server, backends and the per-client model trade-off

The architecture is a websocket server process that accepts audio from clients and pushes transcription results back. Three backends are supported: faster_whisper, tensorrt and openvino. faster_whisper is the default path and the one the installation instructions assume; TensorRT is documented as a separate setup with its own TensorRT_whisper.md and requires you to build engines before starting the server; OpenVINO targets Intel CPUs, iGPUs and dGPUs, with the README noting that the Docker route enables GPU support without extra host configuration while native use requires Intel drivers and the OpenVINO runtime installed and configured.

The most consequential design detail is model instantiation. According to the README, when you run the server without specifying a model, it instantiates a new Whisper model for every client connection. That buys flexibility: each client can request a different model size. It costs you a load wait on connect and higher RAM or VRAM use, and the cost scales with concurrent clients. Passing a custom model through -fw or -trt switches the server to single model mode, instantiated once and reused across connections. If you know your model size up front, single model mode is the configuration you want; the default is friendlier for heterogeneous clients and worse for memory planning.

Installing WhisperLive and running a first server

Installation has a system dependency before any Python work. PyAudio needs PortAudio, and the repository provides a setup script that handles the platform difference: portaudio19-dev on Debian and Ubuntu, portaudio-devel on Fedora, and Homebrew's portaudio on macOS.

bash scripts/setup.sh

Next, create a Python 3.12 virtual environment and install the package from pip. On Fedora the README notes you may need python3.12 and python3.12-pip from dnf first.

python3.12 -m venv whisper_env
source whisper_env/bin/activate
pip install whisper-live

With the environment active, start the server on port 9090 with the faster_whisper backend. The two limits below are worth setting from the start: max_clients defaults to 4 and max_connection_time defaults to 600 seconds.

python3 run_server.py --port 9090 \
                      --backend faster_whisper \
                      --max_clients 4 \
                      --max_connection_time 600

For a file rather than a microphone, the README's OpenAI REST path pairs a server started with --enable_rest and a client script. The server command includes --cors-origins, which matters if a browser page will call it.

python3 run_server.py --port 9090 --backend faster_whisper --max_clients 4 --max_connection_time 600 --enable_rest --cors-origins="http://localhost:8080,http://127.0.0.1:8080"
python3 client_openai.py $AUDIO_FILE

On CPU, thread count is a real knob rather than a detail. OMP_NUM_THREADS defaults to 1, and the --omp_num_threads argument overrides it, so a four-thread server looks like this.

python3 run_server.py --port 9090 \
                      --backend faster_whisper \
                      --omp_num_threads 4

Advanced features that change what you can build

Beyond plain transcription, the README documents word-level timestamps, custom vocabulary or hotwords, speaker diarization, batch inference, and raw PCM input. Hotwords matter more than they look: domain terms, product names and acronyms are exactly what Whisper mangles, and a vocabulary list is the cheapest correction available before you consider fine-tuning. Word-level timestamps are the prerequisite for subtitle rendering and for aligning a transcript to a recording. Raw PCM input is the escape hatch when your audio does not arrive in a container format the server understands; the repository includes examples/manual_audio_chunking.py for the manual chunking path, which the README treats as its own streaming client mode. The browser extensions in the repository show the intended client shape for a Chrome or Firefox capture flow. None of these features are described with performance numbers in the README, so treat them as capability statements rather than measured claims.

Where WhisperLive is the wrong tool

The first limitation is hardware. faster_whisper on CPU is viable but the default OMP_NUM_THREADS of 1 tells you the project expects you to tune rather than assume. TensorRT, the path most likely to give low latency, is also the path with the most setup: the README recommends using the Docker setup for TensorRT and says you must build your TensorRT engines before running the server. That is not a pip install and a flag.

The second limitation is resource behaviour under concurrency. Default mode loads a model per client connection, so a server that comfortably handles two clients can fall over at eight, and the failure mode is memory pressure rather than a clean error. Cap it with --max_clients and know your model size before you expose the port.

The third is that this is not a product. The package classifiers mark Development Status as Beta. There is no desktop application in the repository, no Windows installer, no macOS app bundle. If what you want is to press a key and have text appear in whatever window has focus, WhisperLive is the wrong layer: you would be building the client yourself on top of a websocket protocol. It is also the wrong choice if you need guaranteed transcript stability, since a streaming approach that revises text as more audio arrives behaves differently from transcribing a finished file, and the README does not document how revisions are surfaced to clients.

WhisperLive compared with whisper-streaming

The comparison people search for is whisper-streaming versus WhisperLive, and the difference is architectural rather than cosmetic. WhisperLive is a client-server system: a long-running server process owns the model and the audio pipeline, and clients connect over websockets, which is what lets one GPU serve several users and what makes the browser extensions, the iOS client and the OpenAI-style REST interface possible. whisper-streaming is a library you call from inside your own process, which means no network hop and no server to operate, but also no shared model across users and no built-in transport. If you are embedding transcription into a single Python application on one machine, the library shape is simpler. If you need multiple clients, a remote GPU, or a browser talking to a backend, the server shape is the reason to pick WhisperLive.

Maintenance, upgrades and the MIT licence

WhisperLive is not archived, and the last push to the main branch was on 2026-09-10. Releases have moved at a steady cadence: v0.8.0 on 2026-03-17, v0.9.0 on 2026-06-02, and v0.10.0 on 2026-09-07. That cadence is the upgrade cost you should plan for. The websocket protocol between server and client is the interface most likely to shift between releases, so pin the whisper-live version in your client environment and read the release notes before moving, rather than tracking main. The dependency list in setup.py pins faster-whisper==1.2.0 and constrains onnxruntime by Python version, which means Python version choice affects what resolves; the README asks for Python 3.12 while the classifiers list 3.9 through 3.13.

The project is MIT licensed, which is permissive and imposes no source disclosure obligation on your own code. One practical consequence worth checking rather than assuming: WhisperLive is a wrapper, and the models you load carry their own licences. The README points at Whisper models and at OpenVINO's Hugging Face uploads. If you deploy commercially, confirm the licence of the specific model weights you serve. This is a description of the licence terms, not legal advice.

Editorial conclusion

Adopt WhisperLive if you are building an application that needs streaming transcription and you are willing to run and size the server yourself; the websocket interface, hotwords, diarization and word-level timestamps cover most integration needs without you writing the audio pipeline. Do not adopt it if you want a finished desktop dictation product, or if you cannot install PortAudio and a Python 3.12 environment on the host. Before committing, verify three things on your own hardware: that the backend you intend to use actually builds (TensorRT requires the separate TensorRT_whisper.md setup and an engine built in advance), that your VRAM budget survives the default one-model-per-client behaviour, and that your client library speaks the websocket protocol the version you pinned expects.

Frequently asked questions

Official sources

1. collabora/WhisperLive on GitHub
2. Issues
3. License: MIT
4. README
5. Releases