Vui: a streaming voice loop in one Python server, with an open-core boundary you should read before adopting

The problem Vui targets: assembling a speech loop without writing the plumbing

Building a spoken assistant means stitching together at least four moving parts: audio capture, speech recognition, a language model, and speech synthesis. Each one has its own latency budget, and the failure mode is always the same. The parts work in isolation and the conversation still feels wrong, because the reply only starts after the transcription finishes and the transcription only starts after you stop talking. Vui's answer is a single Python server that owns the whole loop and streams between stages instead of batching them. The README describes the pipeline as ASR to LLM to TTS with VAD-driven turn taking, speculative LLM prefill while you are still speaking, and sentence-level TTS chunking with backpressure. Those three mechanisms are the actual product. Everything else in the feature list is a component choice around them.

The intended user is an engineer who wants a working voice interface on their own hardware and does not want to write WebRTC signalling, VAD, and streaming chunk management from scratch. The repository also ships a one-shot REST endpoint at POST /v1/voice-note that runs the full pipeline in a single HTTP call with audio in and JSON out, which suits a different user: someone who wants voice input in an existing application and does not need a persistent session.

What actually streams: WebRTC in, WebSocket control, sentence-level chunks out

The transport layer is split. WebRTC carries the audio, and a WebSocket pipeline carries the control and event flow, with a browser UI included. Turn boundaries come from voice activity detection rather than a push-to-talk button, and the README states that the LLM prefill starts speculatively while the user is still speaking. That is the mechanism that hides model latency: by the time VAD declares the turn over, part of the prompt is already processed. It is also the mechanism most likely to misbehave, because a speculative prefill on a turn that never ends is wasted compute, and a turn that ends earlier than expected leaves the prefill short.

On the output side, TTS is chunked at sentence granularity with backpressure. Backpressure is the interesting word here. It implies the synthesis stage can signal the text stage to slow down rather than letting audio buffers grow without bound. The README does not document the buffer limits or the queue depth, so the practical behaviour under a fast LLM and a slow GPU is something you would have to observe yourself.

Barge-in is handled the same way it is in most streaming assistants: the user starts talking mid-reply and the model cancels and listens. The README states this plainly without describing how cancellation propagates to the TTS chunk already in flight.

Vui Nano: a 300M speech transformer over the Qwen3-TTS codec

The synthesis model is Vui Nano, 300M parameters, described as a Llama-style decoder with an RQ-Transformer head over the Qwen3-TTS-12Hz codec. The README says it was trained on conversational speech that includes breaths, laughter, hesitations, and multi-speaker dialogue, which is a deliberate departure from clean read-speech corpora. Whether that produces more natural interruptions is a judgement you would have to make by listening; the repository does not publish a comparison against a baseline.

The performance figure in the README is roughly 9x realtime on a 4090 with bf16 inference and CUDA graphs. Treat that as a single-hardware reference point, not a general claim. It says nothing about smaller GPUs, and the Apple Silicon path is quantified separately at roughly 1.5 to 2.7x realtime on an M4 with quantized vui-190k weights and an MLX backend.

Two conditioning features sit on top of the model. Voice cloning takes an uploaded audio sample and reproduces that speaker, with four fine-tuned presets shipped under the names maeve, abraham, rhian, and harry. Separately, SQ and WPS conditioning biases generation across six speech-quality channels and words-per-second. Words-per-second conditioning is the more practically useful of the two: it gives you a dial for pacing without retraining, which matters when a long LLM answer needs to fit a shorter reply window.

Getting it running: the one-liner, the compose file, and the host Ollama assumption

There are two install paths. The one-liner is curl -fsSL https://install.fluxions.ai | bash, which clones into ~/vui, detects Docker versus native, installs dependencies including uv, Ollama, ffmpeg and the Claude Code CLI, pulls the model, and launches on http://localhost:8080. Flags include --docker, --native, --no-claude, --upgrade, --model <name>, and --dry-run, and the README points at ./install.sh --help for the full list.

The recommended path is docker compose, and it comes with a specific assumption worth reading carefully. The intended setup is Ollama on the host plus the Vui container, with the container using host networking to reach localhost:11434. Prerequisites are Docker with the Compose plugin, the NVIDIA Container Toolkit, and Ollama on the host. The README gives the toolkit commands for Debian and Ubuntu (install nvidia-container-toolkit, run nvidia-ctk runtime configure --runtime=docker, restart Docker) and a verification command using nvidia/cuda:13.0.0-base-ubuntu22.04. The stack is described as designed for Linux plus NVIDIA GPU.

If you would rather not run Ollama on the host, the compose file has an ollama profile: docker compose --profile ollama up -d, followed by docker compose exec ollama ollama pull qwen3.5:4b. The host path uses the same model tag, ollama pull qwen3.5:4b. Model weights download from Hugging Face on first run and persist in a named volume. There is also a claude-task compose profile for a sidecar agent that handles slower work through your existing Claude Code MCPs, auto-discovered on boot, and the README notes it can be pointed at Ollama, z.ai, DeepSeek, vLLM, LM Studio or LiteLLM through an Anthropic-compatible /v1/messages envelope.

The open-core boundary, and why the licence metadata does not match the description

This is the part to read before anything else. The README states directly that the repository is the open core, and that the production API ships ongoing model updates and a more advanced turn-taking system on hardened, low-latency infrastructure. Turn-taking is the hardest part of a voice assistant and the feature the README leads with in its own feature list. The open repository therefore contains a turn-taking implementation, but the README positions a better one as commercial. That is a legitimate business model and it is disclosed, not hidden. It does change what you are evaluating: you are not assessing a finished turn-taking system, you are assessing a starting point that the maintainers say they have improved elsewhere.

The second mismatch is the licence. The repository description says Apache 2.0. The metadata field supplied for this repository says NOASSERTION, meaning the licence could not be automatically identified. Those two things can both be true if, for example, the licence file is present but not in a form GitHub recognises, or if the repository contains components under different terms. I cannot resolve it from the material available. If you plan to redistribute Vui or ship it inside a product, read the actual LICENSE file in the clone and check whether the model weights on Hugging Face carry separate terms from the code. Do not treat the description string as the licence.

Where Vui is the wrong tool: Apple Silicon, CPU-only hosts, and the ASR backend choice

The clearest limitation is platform. The compose path is documented for Linux with an NVIDIA GPU, and the README is explicit that on Apple Silicon the Engine Python API auto-dispatches to an MLX backend and that demo.py and demo.py --render work end to end, but the streaming-server MLX glue is WIP. Read that as: the model runs on a Mac, the server does not, at least not as a supported path. If your target is a Mac or a CPU-only box, the streaming assistant is not the part you get today.

The ASR layer is pluggable between faster-whisper on GPU and Moonshine on CPU with ONNX streaming. That choice is not cosmetic. Running Moonshine moves recognition off the GPU and onto the CPU, which frees VRAM for the LLM and the TTS model but changes the latency profile of the whole loop, because recognition now competes with the rest of the host for CPU time. The README does not publish latency numbers for either backend, so the trade-off is something you measure rather than read.

There is also a dependency shape problem. Vui expects an LLM endpoint to exist: Ollama on localhost:11434, or vLLM, or any OpenAI-compatible endpoint. That is flexible, but it means Vui is not self-contained. A deployment that cannot run a second model server alongside it is not a deployment Vui supports. The bundled Ollama profile softens this, at the cost of running two containers and pulling the model inside the compose network.

The alternative worth comparing: Pipecat, and the difference in where the loop lives

The natural comparison is Pipecat, the open source framework for voice and multimodal conversational pipelines. The difference is architectural rather than feature-level. Pipecat gives you a frame-based pipeline abstraction: you compose services (a transport, a VAD, an STT, an LLM, a TTS) into a graph and the framework moves audio and text frames between them. Vui gives you one opinionated server with the components already chosen and wired, plus a browser UI and a compose file that brings the stack up. Pipecat's approach means you can swap any stage for a service the framework supports, including cloud APIs, and you own the composition. Vui's approach means you get a working loop faster and you accept its choices, or you fork it.

That distinction decides the adoption question. If your requirement is a specific STT vendor, a specific cloud TTS voice, or a transport Vui does not implement, Pipecat is the better starting point because the pipeline is the product. If your requirement is a local, self-hosted assistant on one GPU with the OpenAI Realtime API surface, Vui has already made those choices and the README documents the compatibility path in docs/realtime-api.md. Note that Vui's OpenAI compatibility is a server-side surface for clients written against that spec, which is a narrower claim than being a drop-in replacement for the hosted service.

Maintenance cost and what the repository tells you about it

Vui is young. The releases listed are v0.1.0 (Vui 100M) and v1.0.0 (Vui Nano plus streaming voice assistant), both dated 2026-05-14, with the most recent push to main on 2026-09-02. That is roughly four months of activity after the 1.0 tag, which is not enough history to judge how the maintainers handle breaking changes in the WebSocket protocol or the compose file.

The upgrade surface is larger than a typical Python library. You are tracking the Vui checkpoint and the Qwen codec on Hugging Face, the Ollama model tag (qwen3.5:4b in the documented examples), the faster-whisper or Moonshine ASR backend, the NVIDIA Container Toolkit version, and the CUDA base image referenced in the verification command. The install script has an --upgrade flag, which suggests the maintainers expect in-place upgrades to be a normal operation, but the README does not describe what --upgrade preserves. Memories are persisted to ~/.vui/memories.json, which is outside the container and outside the named model volume, so it is worth confirming where that file lives in your Docker setup before you upgrade anything.

The claude-task profile adds a second maintenance axis. It depends on the Claude Code CLI and your existing MCPs, and the README says it is auto-discovered on boot. Auto-discovery is convenient and it is also the kind of behaviour that changes silently when the CLI changes. If you do not need agentic task handling, the --no-claude install flag and the absence of that compose profile are the simpler configuration.

Editorial conclusion

Adopt Vui if you want a self-hosted, OpenAI Realtime API compatible voice loop on Linux with an NVIDIA GPU, and you accept that the repository describes itself as the open core while production turn-taking lives in the hosted service. Skip it if you need a finished streaming server on Apple Silicon (the README calls the MLX streaming glue WIP) or if you cannot run a separate Ollama or vLLM endpoint. Before committing, verify three things: what the actual licence file contains, since the repository metadata reports NOASSERTION while the description says Apache 2.0; whether the bundled claude-task profile is something your deployment can depend on; and whether the turn-taking quality in this repo is enough for your use case, because the README explicitly reserves the more advanced turn-taking system for the production API.

Official sources

1. fluxions-ai/vui on GitHub
2. Issues
3. Project website
4. README
5. Releases