z-waif: a local AI companion that drives a VTube Studio model from a Discord chat

The project is a front end, not a model

z-waif describes itself as a fully local and open source AI Waifu, and the first thing to understand is that the intelligence comes from somewhere else. The README states that it uses Oobabooga, RVC, and Whisper to run the AI systems locally, and that it works as a front end to tie many programs together into one cohesive whole. The goal, in the project's own framing, is less about giving an all in one package and more about giving you the tools and knowledge to create your own AI Waifu. That is a meaningful design choice. You supply the language model server, the voice model, and the speech recognition, and z-waif supplies the character layer, the memory, the chat surface, and the automation. The consequence is that the install is a supply chain of separate programs rather than a single installer, and the documentation reflects that.

The hardware requirement is the first filter

The README is unusually direct about this. The recommended configuration is Windows 10 or 11 with a CUDA GPU from Nvidia with at least 16GB of VRAM. It then says that Mac and Linux are now supported, and that any brand of GPU, whether AMD, Nvidia, or Intel, works with a bare minimum of 8GB of VRAM. That is a real constraint on who can run this. The voice pipeline alone is a speech to text model, a language model, and a voice conversion model, and at inference time they want the GPU. A machine with integrated graphics and 8GB of shared system memory is not in scope. The project is aimed at individuals and hobbyists with a spare gaming PC, which matches the stated emphasis on personal use and companionship.

Conversation is Whisper, RVC, and a model server talking to each other

The feature list describes a speak back and forth loop using Whisper text to speech, which appears to be a slip in the README for text to speech, and the ability to configure your own character's voice with thousands of possible models through RVC. The README also promises a large number of quality of life tweaks without enumerating them. The upstream tools remain separate and configurable in their own right, and the README says the project inherits all the options and modularity from any external software used, Oobabooga and RVC Voice among them. In the repository layout this shows up as a `requirements.txt` that pins a very large set of packages, including the model runtime and the desktop automation stack:

faster-whisper==1.1.1
gradio==4.36.1
ollama==0.5.1
pyvts==0.3.3
discord.py==2.4.0
keyboard==0.13.5
PyAutoGUI==0.9.54

Those seven lines tell you most of the architecture. `faster-whisper` handles speech recognition, `gradio` and `fastapi` provide the web UI and API surface, `ollama` is the local model client, `pyvts` is the VTube Studio integration library, `discord.py` is the chat surface, and `keyboard` with `PyAutoGUI` is the desktop automation layer for hotkeys and mouse control.

Memory is a lorebook plus a RAG over old conversations

The memory story has two parts and it is the most interesting design decision in the project. The first is lorebook entries, which the README describes as letting your character remember a wide array of information as needed, the standard technique from roleplay front ends where a fact is pulled into context only when the topic comes up. The second is the custom RAG, which gives the character knowledge of older conversations, and there is an import path for old logs and conversations so you can keep the same character you had in other software. The v1.15 release added rolling summaries on top of this. A summary is generated every ten message pairs, spaced back by ten message pairs so that things still in live memory are not summarized, and the behaviour is toggleable through the environment, with the total messages summarized per chunk set by `SEARCH_POINT_SIZE` and the maximum size of a summary set by `SUMMARY_TOKENS_MAX_COUNT`. The author warns in the same entry that the RAG should probably be on and might break if it is not, and suggests turning it on after a few days of use.

The character is a VTube Studio model that reacts to what happens

The VTuber integration uses VTube Studio and any compatible models, and the README describes the ability to send emotes to the model based on their actions, along with idle and speaking animation states. `pyvts` in the dependency list is the Python library for talking to that application, and it is worth noting that this makes z-waif a front end for a model you already own rather than a character creator. The README's Diaspora section points out the original project, TumblerWarren's Virtual_Avatar_ChatBot, and credits it as the skeleton this code is spun off from, noting that the original has more of a focus on non-local AI. That is a useful pointer for anyone who wants the same experience with a hosted model instead, and it also establishes the lineage of the codebase.

Discord, vision, alarms, and Minecraft are the modules

The modularity section lists the built in modules. Discord is there for messaging, and v1.15 added new Discord API commands, with `/help` listing all commands, `/allow_newlines` and `/disallow_newlines` toggling the newline cuts, and the `/reroll` command retained. Vision is a module that enables multimodal so the character can see, and the changelog notes a practical tip for the image API, to turn on Multiple-access in the Windows camera settings so a preview of what is being taken can be kept open, which the author says also improves image quality. Alarm is a module so the character can wake you in the morning. Minecraft is the outlier, allowing the character to control the game using Baritone, Wurst, and other command based mods. The README also notes that all of these inherit their options and modularity from the external software they drive.

Installation is a Google Doc and a YouTube playlist

There is no install command in the README, and the project is upfront about where the instructions live. The documentation is a Google Doc that the README says shows how to install, how to use the program, and what options you have, and there is a YouTube playlist for the install. The repository itself ships `startup.bat` and `startup.sh` launchers, a `startup-Install.bat`, a `Configurables/` directory for settings, an `API/` directory, a `RAG_Database/` directory, `Backups/`, `Logs/`, and an `OOBA_Presets/` directory. The environment is driven by a `.env` file, which matches the changelog note that the rolling summaries are toggleable in env and that model configs are changed in Configurables. Support runs through GitHub issues, with the maintainer's email listed in the README, and there is a small community Discord. The licence is a custom one that GitHub cannot classify, so the LICENSE file at the root is the version to read.

Editorial conclusion

z-waif fits a person who wants a locally running AI character with a VTube Studio model, a voice, and long term memory, and who is willing to assemble the Oobabooga, RVC, and Whisper side of it themselves. It is the wrong tool if you need a product with support behind it, since the documentation lives in a Google Doc and the install is a set of video tutorials, and it is the wrong tool on a machine with less than 8GB of VRAM, since the README states that as a bare minimum. Before you start, read the v1.15 changelog entries on rolling summaries, because the note that the RAG should probably be on and might break if it is not tells you the memory system is the fragile part of the stack.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. SugarcaneDefender/z-waif on GitHub