voice-chat-ai: a self-hosted voice front end for Ollama, OpenAI, Anthropic and xAI

The gap voice-chat-ai fills between a chat model and a microphone

Most voice assistants bundle three decisions into one product: who transcribes you, which model answers, and which voice speaks back. voice-chat-ai separates them. The README describes a project where you can run everything locally, use OpenAI for chat and voice, or mix the two, for example ElevenLabs voices driving Ollama models, all controlled from a web UI. That mix-and-match is the actual product. The chat side accepts OpenAI, xAI, Anthropic or Ollama language models. The speech side accepts Spark-TTS, OpenAI TTS, ElevenLabs, Kokoro TTS or Typecast. Transcription defaults to OpenAI, with Local Faster Whisper as an option. The intended user is someone who already runs a local model server and wants a spoken interface on top of it without handing the whole pipeline to one vendor. The README also points at two narrower use cases in docs/games.md and docs/stories.md: interactive games with game master characters, and story adventures. Those are content modes layered on the same voice loop, not separate applications.

How the pipeline is assembled: FastAPI, WebSockets, WebRTC and a character folder

The repository layout shows a FastAPI and uvicorn web layer with websockets, plus aiortc and av for WebRTC. That matches the two conversation modes the README describes. The standard mode records audio, transcribes it, sends text to the chosen chat provider, then synthesizes speech from the reply. The OpenAI Realtime mode uses WebRTC instead, which is what allows the documented behaviour of interrupting the AI and getting instant responses. Characters live in the characters/ directory, and the README notes that the sample .wav files there are used for voice cloning, so a character is effectively a prompt plus a reference voice sample. Sentiment analysis is part of the loop: the README states the app analyzes user mood and adjusts AI responses accordingly, which is why textblob, langid and spacy appear in requirements.txt. Configuration is environment-driven through a .env file loaded at startup, with .env.sample as the template. The web UI is the recommended surface because it lets you change characters, model providers, speech providers and voices on the fly; cli.py exists for terminal use. One design consequence worth naming: because providers are selected at runtime rather than compiled in, a misconfigured provider fails at conversation time, not at startup.

Installing voice-chat-ai locally with Python 3.11

The README requires Python 3.11+, ffmpeg, a microphone, and at least one chat provider and one speech provider. The steps below follow the README and requirements.txt exactly. Note the ordering constraint in requirements.txt: PyTorch must be installed before the core dependencies, and the index URL differs by hardware.

git clone https://github.com/bigsk1/voice-chat-ai.git
cd voice-chat-ai
python -m venv venv
source venv/bin/activate

On Windows the README gives `venv\Scripts\Activate` instead. Next, pick one PyTorch line. The CPU variant:

pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cpu

Or CUDA 12.4:

pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cu124

Then the core dependencies and ffmpeg. The README suggests winget on Windows, apt on Linux and brew on macOS, and says to verify with `ffmpeg -version` after restarting the terminal.

pip install -r requirements.txt

For local voice cloning there is a separate script, documented as optional if you only use OpenAI, ElevenLabs or Kokoro speech. It needs roughly 5GB of disk for the model and works on CPU or a CUDA GPU.

python setup_sparktts.py

If you prefer containers, docker-compose.yml pulls bigsk1/voice-chat-ai:latest, loads .env via env_file, and publishes port 8000. The audio wiring is the part to read carefully: the default environment sets PULSE_SERVER=/mnt/wslg/PulseServer and mounts \\wsl$\Ubuntu\mnt\wslg, which targets WSL2. Native Ubuntu and Debian users are expected to uncomment the pulse cookie and socket lines instead. Kokoro is not bundled. The README tells you to run it from the Kokoro-FastAPI repository, for example in Docker, then point the app at it:

KOKORO_BASE_URL=http://localhost:8880/v1
TTS_PROVIDER=kokoro
KOKORO_TTS_VOICE=af_bella

The README lists am_onyx as the male default and af_bella as female. After that, start the app, open the web UI on port 8000, choose a character and a provider pair, and speak.

Where voice-chat-ai gets expensive or simply wrong

The first limitation is that nothing here is turnkey. You supply a chat provider, a speech provider, ffmpeg, and on Windows a working WSL2 audio path. The docker-compose file encodes one specific audio assumption as the default, and the README does not document what happens when that assumption is wrong beyond the commented alternatives. The second is dependency weight. requirements.txt pins numpy below 1.28.0 and fixes transformers, spacy, faster-whisper and the provider SDKs at exact versions. That is good for reproducibility and bad if you want to share a virtualenv with another project. Third, Local Faster Whisper downloads a model of about 1GB on first use, stored in the user cache directory and shared across environments, so the first transcription after switching to it is not instant. Fourth, the project is clearly built around hosted realtime conversation: the interruptible, instant-response experience the README highlights depends on the OpenAI Realtime API over WebRTC. If your goal is a fully offline, interruptible conversation, the documented path is the standard record-transcribe-reply loop, which is a different interaction. Finally, the README does not document a rollback procedure between releases, and the repository carries no changelog file at the top level, so pinning a known-good version before upgrading is your own responsibility.

Choosing between voice-chat-ai and a hosted voice assistant

The obvious alternative is a hosted assistant such as ChatGPT's own voice mode, and the difference is not quality, it is control. A hosted assistant gives you one fixed chain of models with no configuration file and no local storage. voice-chat-ai gives you a .env file, a characters directory you can edit, and the ability to point transcription at Local Faster Whisper so audio never leaves the machine. The cost is that you own the failure modes: audio device routing, model downloads, provider keys and version pins. A second comparison is against wiring the pieces yourself. The individual components here are ordinary, a FastAPI app, faster-whisper, a TTS client, aiortc. What the project adds is the glue: character definitions with reference .wav files, the sentiment step that adjusts responses, the game and story modes in docs/games.md and docs/stories.md, and a UI that switches providers mid-session. If you only need one provider pair and never change it, that glue is less valuable and a small script may be enough. If you want to audition several voices against several models without rewriting the loop each time, the glue is the reason to use this project.

Licence, maintenance and what an upgrade actually costs

The project is MIT licensed, both in the repository metadata and in pyproject.toml, which permits commercial use and modification provided the copyright notice and licence text are retained. That covers this codebase only. The speech and chat providers you connect to have their own terms, and running Spark-TTS or Kokoro means pulling additional models and repositories under their own licences, which the README links but does not summarize. Nothing here is legal advice; check each provider's terms before shipping anything. On maintenance, the last push to the default branch was on 2026-09-02, and the most recent tagged release is v1.4.0 from 2026-08-01, following v1.3.0 in May 2026 and v1.2.0 in April 2026. The repository is not archived. Upgrade cost is dominated by the pinned dependencies rather than the application code: because requirements.txt fixes exact versions of transformers, spacy, numpy and the provider SDKs, a release that bumps one of them can force a full reinstall of the virtualenv. The practical approach is to install into a dedicated environment, as the README's venv and conda instructions already imply, and to keep the previous environment until a new version has produced a spoken reply end to end.

Editorial conclusion

Adopt voice-chat-ai if you want a self-hosted voice loop where the chat model and the speech provider are separate choices, and you are comfortable installing PyTorch, ffmpeg and Python 3.11 yourself. Do not adopt it if you want a zero-install browser voice assistant or need a documented rollback path between releases. Verify first that your chosen provider set is one the .env.sample actually exposes, that PULSE_SERVER in docker-compose.yml matches your host audio setup, and that you can live with the pinned dependency versions in requirements.txt.

Frequently asked questions

Official sources

1. bigsk1/voice-chat-ai on GitHub
2. Issues
3. License: MIT
4. README
5. Releases