AIwaifu: a self-hosted AI VTuber stack with a VITS voice and a Pygmalion brain

What the project is, and what it wants to be

aiwaifu is an Open sourced finetunable customizable simpable AI waifu inspired by neuro-sama. The stated goal is to just giving everyone a foundational platform to develop their own waifu, and it is powered by opensource AI models for self-hosted or deployed use.

The word foundational is doing real work in that sentence. This is not a character you install and talk to. It is a starting point that you clone, retrain and reshape, with the README's own framing being YOUR ONE AND ONLY WAIFU if you have your own datasets or custom the personality. The fine-tuning is the point.

That framing also explains the project boundaries. The code of conduct section states that everything made is open sourced, free and customizable to the very core, and that proprietary models will never be included because they conflict with that and because they censor the content the project is built around.

So the stack is three known open models wired together, and the contribution you make is the fine-tuned weights and the personality prompt on top of them.

The project is by HRNPH, the primary language is Python, the licence is GPL-2.0, the default branch is main, and the last push was on 2026-07-07. There are no GitHub releases, so the project is distributed as source. The topics name the territory: ai-waifu, anime, artificial-intelligence, vtuber, waifu, and hacktoberfest.

The README is written in a deliberately casual register, with a video of the character in action linked from the What is it section, and the badges at the top are the forthebadge set including a 0 percent optimized one. The tone is part of the project's identity, and the technical content underneath it is specific enough to be useful.

The three models, and why the output is in Japanese

The Quicknote section names the models, and the choices explain a lot about how the character behaves.

The current TTS model is a VITS pretrained model, hosted on Hugging Face, and the README notes that this may change later for a more customizable option. So the voice is a VITS model, and the avatar mouth movement is driven by the same audio through VTube Studio.

The language model is Pygmalion1.3b, which is a fine-tunable model in the 1.3 billion parameter range.

And then the third one, which is the most interesting technical decision in the project. The reason TTS was in Japanese is, in the README's words, because it is cuter, and the model outputs from English to Japanese are translated using the Facebook nllb-600m model.

So the pipeline is Pygmalion produces English dialogue, NLLB translates it to Japanese, and VITS speaks the Japanese. That is an unusual arrangement, and it has an obvious consequence worth thinking about: every reply goes through a translation model, which means the character speaks in a language the underlying language model was not tuned for, and the translation step is a place where nuance can be lost or where the same English phrase can be rendered in several ways.

The model stack in the pyproject backs this up. The dependencies include torch at 2.0.1, transformers at 4.30.2, librosa at 0.10.0.post2, soundfile, pydub, soxr, sentencepiece, spacy-legacy and its loggers, and huggingface-hub at 0.16.4, plus datasets at 2.13.1 and tokenizers at 0.13.3, which are the pieces you need to prepare and fine-tune a model rather than merely run one.

The requirements file pins the same versions more exactly, with torchaudio at 2.0.2, uvicorn at 0.22.0 for the server, websocket-client at 1.6.1 and websockets at 11.0.3, pyyaml, pydantic and pypinyin, which is a Chinese text conversion library and a clue about the character's language handling.

The project name in the pyproject is aiwaifu at version 0.1.0, described as a foundational ai waifu development platform for any developer, with the author contact at a protonmail address.

The server and client split, and the reason it exists

The design decision that explains most of the rest is the split between an inference server and a client, and the README explains it in a parenthetical that gives the real reason.

The software was split into an HTTP server for model inference, since the author needed to use a home server because the model takes too much RAM, with RAM rather than VRAM, more than 12 GB required and 16 GB recommended.

So the model runs on one machine and the character client runs on another, talking over HTTP. That is a sensible arrangement for a project whose whole point is that you own the hardware, because the machine that holds the weights is often a server and the machine with the microphone and the avatar is often not the same one.

The server is started with a single command:

# this run on localhost 8267 by default
python ./api_inference_server.py

And the client is a separate process:

# this will connect to all the server (Locally)
# it is possible to host the api model on the external server, just beware of security issue
# I'm planning to make a docker container for hosting on cloud provider for inference, but not soon
python ./main.py

The two comments carry the operational warnings. The client connects to the server, and it is possible to host the model on an external server, but the README warns to beware of security issues, and the author notes that a container for cloud hosting is planned but not soon. So in the current state, exposing that inference port to a network is on you.

Then the last step of the setup is to open VTube Studio and allow access, because the client needs the avatar application to accept its requests.

The repository layout matches this split. There is api_inference_server.py and main.py at the root, a vtube_studio.py module for talking to the avatar application, a Conversation/ directory for the dialogue logic, an AIVoifu/ directory holding the voice conversion and model code including the SoVITS monotonic_align module, an audio_cache/ directory, a pyproject.toml and a poetry.lock, and test_chat.py and test_sound.py for trying the pieces individually.

That last pair is worth mentioning to anyone evaluating the project. Having separate test scripts for the chat path and the sound path means you can check whether the language model answers and whether the voice synthesises without wiring up the avatar at all.

Installing it, and the pieces that are easy to miss

The installation section lists the prerequisites before the commands, and each one corresponds to something that will fail confusingly if it is missing.

You need Python 3.8.X and more than 10 GB of storage with a decent internet connection for the model weights. You need C and C++ build tools with CMake installed, with a link to an issue comment for the details if you do not. And you need Git LFS installed to handle the large file downloads in git.

The Git LFS point is the one that catches people, because the weights are stored in the repository and a plain clone without LFS will produce pointers instead of files.

Then the clone:

git clone https://github.com/HRNPH/AIwaifu.git
cd ./AIwaifu

For dependencies there are two routes, and the README marks one as recommended. With Poetry, you install Poetry, then:

poetry install
poetry shell # to activate the virtual environment (need to do this everytime you want to use this projects)

The alternative, marked not recommended, is a plain virtual environment with the requirements file:

# may contain some bloated packge(since I didn't clean the requirements YET)
# so I recommend install this on venv

# ---- optional -----
python -m venv venv
./venv/script/activate  # for windows
# source ./venv/bin/activate # for linux
# --------------------

# you need to uninstall websocket module and install websocket-client (which was included in the requirements for it to work)
pip uninstall websocket
pip install -r ./requirements.txt

The websocket swap is a real trap rather than a quirk. The websocket module and the websocket-client package are different things with confusingly similar names, and the README tells you to uninstall the first and install the second because the requirements include it for the code to work.

Then there is a native build step that is easy to miss because it is not a pip install:

# You need to install the monotonic_align module for sovits to work
cd AIVoifu/voice_conversion/Sovits/monotonic_align
python setup.py build_ext --inplace && cd ../../../../

That compiles a Cython extension in place for the SoVITS voice conversion code, which is the part of the stack that has to build rather than install. The directory name and the build_ext call are both in the README, and the cd back out at the end is part of the same command so you end up where you started.

The final piece is the avatar side. You download and start VTube Studio from Steam, install the VTS desktop audio plugin by Lua Lucky following the linked video, and then open the plugin API at port 8001 in the application settings, or another port if you change the code.

The hardware, and what the numbers actually mean

The Minimum Specs section is short and it is the most decision-relevant part of the README.

For CPU only you need 12 GB of RAM or more, and the acceptable runtime was tested on an i7 7700k at 1.00 to 2.30 minutes, with a note that a faster CPU would be better. For inference with a GPU you need a minimum of 8 GB VRAM, with the note that it will take at least 7.2 GB of VRAM, that it is Nvidia GPU only, and that it is very fast on a K80, which was the tested card, with a faster or equivalent GPU being pretty fast too.

The 1.00 to 2.30 minute figure is the one to look at. That is the time for a response on a CPU-only machine from 2017, and it tells you what the experience is like without a GPU: the character takes a minute or two to answer, which is a very different feel from a streaming avatar that replies in a second.

The 7.2 GB of VRAM figure is also useful in a practical way. It is stated as a hard floor separate from the 8 GB minimum, and it tells you the model plus the inference runtime has to fit in about seven gigabytes of video memory, which excludes a 4 GB card and fits comfortably on an 8 GB one.

The K80 reference is a useful signal too. A K80 is an older datacentre card, so the tested configuration was not a consumer GPU, and the author is saying that ordinary hardware of a similar or better class will be fast. The requirement of Nvidia only is because the CUDA path is what is being measured, and there is no CPU or AMD equivalent in the numbers.

Combined with the server and client split, the hardware picture is coherent. The inference needs the memory, the client needs a microphone and a display, and if you do not have a discrete card you can still run the whole thing on a machine with 12 GB of RAM, at the cost of a minute-plus latency per reply.

What the project asks of you, and how it handles its own rough edges

The closing sections of the README set the tone for contributing, and they are worth reading as a description of what kind of project this is.

The How Can I help section asks you to take a look at the issues, ask questions, suggest new features or tests, and to push the model to its limit. There is a link to a discussions thread specifically for commenting on model performance, which is a sensible place to compare outputs.

The project also has a Discord and a star history chart, and the badges at the top include a check it out one alongside the built with love and makes people smile ones. The informal register is consistent throughout.

Then there is the line that most honestly describes the state of the code. Sometimes things can be broken, especially in the server, and if you happen to find what is broken feel free to open an issue or a pull request.

That is a fair summary of what a fine-tuning platform built around a third-party voice model and a third-party language model is going to be like. The integration points are where the breakage lives, not the model code, and the author is telling you that in advance rather than pretending otherwise.

The .gitignore, the AIVoifu directory, the Conversation directory, the audio_cache directory and the image directory complete the top level, and the presence of an audio_cache tells you that generated speech is written to disk rather than streamed straight to the avatar, which is a reasonable trade for keeping the VTube Studio audio path simple.

One thing the README does not offer is any statement about who the character is or what the personality contains, because the personality is the thing you supply. The defaults exist so the system runs end to end on first launch, and the fine-tuning path is where the project wants your effort.

Editorial conclusion

Adopt AIwaifu if you want an AI character you can fine-tune on your own data and run entirely on your own hardware, and you are willing to put a home server, a VTube Studio model and about twelve gigabytes of memory into it. Do not adopt it expecting a plug-and-play product, because the setup involves compiling a native extension, a VTube Studio plugin and a two-process client and server split. Before you start, read the hardware section, because the CPU path needs 12 GB of RAM and the GPU path needs at least 7.2 GB of VRAM on an Nvidia card, and note the README's own warning that the server side sometimes breaks.

Frequently asked questions

Official sources

1. HRNPH/AIwaifu on GitHub
2. Issues
3. License: GPL-2.0
4. README