VoiceStreamAI: self-hosted Whisper transcription over WebSocket

The problem VoiceStreamAI addresses

Whisper is a batch model. You hand it a finished audio file, it hands back text. Live audio does not arrive that way, and calling Whisper on every 100 ms of microphone input is wasteful: most of those windows contain silence, a cough, or a chair scrape. VoiceStreamAI exists to sit between a microphone and Whisper. A Python server accepts a WebSocket connection, receives audio chunks, runs a Voice Activity Detection model to decide which parts are speech, and forwards only those parts to an ASR pipeline. The client is a plain HTML/JavaScript page that captures microphone audio and displays transcripts.

The intended user is a developer who wants transcription on hardware they control. The README frames the stack as self-hosted and modular: VAD and ASR are both replaceable through a factory and strategy pattern, and the server takes command line arguments for which implementation to load. There is no hosted service, no API key beyond the Hugging Face token that Pyannote requires, and no vendor account. That matters if your audio cannot leave your network, or if you want to swap the model without rewriting the transport layer.

How the server, VAD and ASR fit together

The repository is split into src/ for the server and client/ for the browser page, with test/ alongside them. The server owns the WebSocket lifecycle and the chunking logic. The README says new processing and chunking strategies can be added in server.py and selected by a client through the processing_strategy key in its config, which means the client, not the server, decides how audio is cut up.

Three VAD options are exposed through --vad-type: silero, pyannote, or none. Silero is described as the token-free default, so a fresh install does not need a Hugging Face account. Pyannote requires an auth token, supplied either in --vad-args as a JSON string or through the PYANNOTE_AUTH_TOKEN environment variable in the Docker path. The ASR side defaults to faster_whisper, and --asr-args accepts a JSON object that can change model_name or model_size. The README also mentions OpenAI's openai/whisper-large-v3 as an alternative ASR target.

The reason VAD sits in the middle is stated plainly: it reduces compute by skipping non-speech, improves accuracy by not feeding noise to the recognizer, and cuts bandwidth by sending only voice segments. The README also gives a number worth reading twice. It reports that the speech recognition pipeline takes 7 seconds on a Tesla T4 with 16 GB, and tells you to take that into account when setting chunk length. That is the central design tension of the project: a streaming interface in front of a recognizer whose single pass is measured in seconds.

Installing VoiceStreamAI and running a first transcription

The README offers two install paths. The manual one needs Python 3.8 or later and the packages pinned in requirements.txt, which include websockets, faster-whisper, pyannote.audio, silero-vad and torch. The Docker path builds on nvidia/cuda:12.2.2-cudnn8-runtime-ubuntu22.04 with Python 3.10 and exposes port 8765.

For a manual install, the README gives this single command after the dependency list:

pip install -r requirements.txt

Once the dependencies resolve, the README's token-free GPU example starts the server on all interfaces with a small Whisper model. Note that this example uses port 6006, not the default 8765:

python3 -m src.main --host 0.0.0.0 --port 6006 \
  --asr-args '{"model_size": "tiny"}'

The module is invoked as src.main, and --asr-args takes a JSON string, so the quoting above matters. If you want Pyannote VAD instead of Silero, the README says to obtain a token from pyannote/segmentation on Hugging Face and pass it in:

python3 -m src.main --vad-type pyannote \
  --vad-args '{"auth_token": "vad token here"}'

For the container route, build the image and run it with the GPU flag, the port mapping and the token:

sudo docker build -t voicestreamai .

sudo docker run --gpus all -p 8765:8765 -e PYANNOTE_AUTH_TOKEN='VAD_TOKEN_HERE' voicestreamai

The README notes that adding a named volume for /root/.cache/huggingface stops the models being re-downloaded on every container start. The Dockerfile's entrypoint is python3 -m src.main with a default command of --host 0.0.0.0 --port 8765, so the container listens on 8765 unless you override it. The README also warns that it does not walk you through CUDA in Docker in detail.

On the client side there is nothing to build. Open client/index.html in a browser, enter the WebSocket address (the README gives ws://localhost:8765 as the default), set the audio chunk length and offset, pick a transcription language, click Connect, then Start Streaming. The README does not state what the chunk length or offset defaults are, so expect to experiment. Run python3 -m src.main --help to see the full argument list.

The latency budget is the real constraint

The README's own figure is the thing to plan around: 7 seconds for a speech recognition pass on a Tesla T4 with 16 GB. A near-realtime transcript built on a multi-second pass is not the same product as a token-by-token streaming recognizer. Whatever chunk length you choose interacts with that number, and the README explicitly tells you to factor it in. If your use case is live captions where a two-second lag is unacceptable, this architecture will not get you there by configuration alone.

There is a second cost the README does not quantify: the VAD model itself is a neural network, and the Pyannote path gates it behind a Hugging Face token and a model download. The Docker image pulls a CUDA base and installs torch, torchaudio and torchvision, so the image is not small and the first run downloads models unless you mount the cache volume. The requirements file pins exact versions in several places (websockets==12.0, faster-whisper==1.0.2, silero-vad==5.1.2, transformers==4.40.2) alongside range pins for the pyannote family, which means dependency resolution is constrained but not frozen. The README does not document a supported upgrade path between versions, and the repository has no releases, so there is no changelog to consult before you move a pin.

Finally, the README's configuration surface is thin on defaults. Chunk length, offset, processing strategy and language are all client-side choices, and the README describes them without specifying recommended values. That is a design decision with a cost: you are expected to tune the system to your audio and hardware rather than inherit a working preset.

When to pick something else

The closest point of comparison is a client-side or browser-native recognizer such as the Web Speech API. The difference is architectural rather than cosmetic. A browser API runs recognition inside the browser, sends nothing to your server, requires no GPU, and needs no Python environment. VoiceStreamAI sends audio over a WebSocket to a Python process that runs VAD and Whisper, which is what lets you choose the model, keep the audio on your own network, and get multilingual transcription from a checkpoint you control. If your only requirement is dictation in a supported browser, the browser API is less machinery for the same visible result. VoiceStreamAI earns its place when the model choice, the data path, or the language coverage has to be yours.

The other comparison worth making is against running Whisper yourself on recorded files. That is simpler: no WebSocket, no VAD, no chunking strategy, no client page. It also cannot produce a transcript while someone is still speaking. VoiceStreamAI's contribution is the streaming layer and the VAD gate in front of the model, not the model itself. If you already have a batch transcription pipeline and your users are fine waiting for the file to finish, adding this server buys you nothing.

Licence, maintenance and what to check before adopting

VoiceStreamAI is MIT licensed, and the repository carries LICENSE.txt. MIT is permissive: you can use, modify and redistribute the code, including commercially, provided the copyright notice and licence text are retained. That is a statement about the project's licence, not legal advice about your situation, and it says nothing about the models you load. Faster-whisper, the Silero VAD package and the Pyannote models are separate works with their own terms, and the Pyannote path additionally requires accepting conditions on Hugging Face to obtain a token. Check those terms separately before shipping.

On maintenance, the last push to the default branch was on 2026-08-13. There are no releases in the repository, so there is no versioned artefact to pin against and no published upgrade notes. Dependencies are pinned in requirements.txt, which means an upgrade is a manual edit followed by a reinstall, and the README documents no rollback procedure. The Dockerfile pins the CUDA base image to 12.2.2 and Python to 3.10, so moving to a newer CUDA runtime means editing the Dockerfile yourself. Budget for that: this is a project you adopt as source, not as a package with a release cadence.

Editorial conclusion

Adopt VoiceStreamAI if you need an MIT-licensed transcription server you can run on your own GPU and you are willing to tune chunk length against the transcribe time the README reports. Do not adopt it if you need streaming output tokens, a managed endpoint, or a project with releases and a documented upgrade path; there are no releases in the repository and the README documents no migration or rollback procedure. Verify first that your GPU can beat the 7-second-per-pass figure on a Tesla T4, that you have a Pyannote token if you want pyannote VAD, and that your client can tolerate the chunking delay the selected processing strategy implies.

Frequently asked questions

Official sources

1. alesaccoia/VoiceStreamAI on GitHub
2. Issues
3. License: MIT
4. README