AI-Waifu-Vtuber: a Python pipeline that turns your voice into a talking VTube Studio avatar

What AI-Waifu-Vtuber actually is, and who it is for

This is a single-user desktop pipeline. It listens on your microphone, transcribes what you say with Whisper, sends the transcript to OpenAI in the persona you wrote, optionally translates the reply, synthesises speech, and plays that audio so VTube Studio can lip-sync to it. A second mode reads Twitch chat through pytchat or the IRC endpoint and answers viewers instead of you.

The README frames it as an AI waifu virtual YouTuber, and the repository layout matches that framing: run.py is the entry point, characterConfig/Pina/identity.txt holds the persona, speaker.json holds VoiceVox speaker ids, and chat.txt and output.txt exist purely so OBS can display captions. The intended user is a streamer who is comfortable editing Python files by hand, not someone looking for a packaged application. There is no installer, no configuration UI and no command line flags documented in the README. Everything is a source edit.

The data flow: microphone, Whisper, OpenAI, translator, TTS, virtual cable

The chain is linear and each stage writes a file or a variable the next stage reads. Audio is captured into input.wav. The transcribe_audio function opens that file and calls openai.Audio.transcribe("whisper-1", audio_file) to get text. The transcript is prefixed with the owner name and appended to a conversation list as a user message, which openai_answer() then sends to the chat model.

If VoiceVox is the output, the model's answer must be Japanese, so a translation step runs first. The README shows two interchangeable calls, translate_deeplx and translate_google, both taking a detected source language and "JA" as the target. The translated string goes to voicevox_tts or silero_tts, and the resulting audio is written to output.wav. The final hop is outside the code: you route desktop audio through VB-Cable and select that cable as the microphone inside VTube Studio, which is what makes the avatar's mouth move.

The design consequence is that latency stacks. Whisper transcription, the OpenAI round trip, translation and synthesis all happen before a single frame of lip-sync. Nothing in the README claims a latency figure, and there is no queue or streaming path visible in the described code.

Installing AI-Waifu-Vtuber and getting one spoken reply

The README gives a flat dependency install and a config file you create yourself. Run the install from the repository root, then create config.py with your OpenAI key and an owner name. Note that requirements.txt pins openai==0.28.1 deliberately, with a comment explaining that the legacy openai.Audio and openai.ChatCompletion calls were removed in openai 1.0 and that newer 1.x releases pull an httpx version that conflicts with googletrans==4.0.0rc1.

pip install -r requirements.txt

api_key = 'yourapikey'
owner_name = "Ardha"

Next, edit the persona file. The README points at characterConfig/Pina/identity.txt and says the answer language from OpenAI depends on the lore language you write there, so a Japanese identity is what makes the VoiceVox path coherent. If you stream, the README also shows a blacklist list in run.py, with Nightbot and streamelements as the given examples, to keep bots out of the conversation.

blacklist = ["Nightbot", "streamelements"]

For the Japanese voice path, start a VoiceVox engine before running the program and confirm the URL the code uses. The README states you can run it locally with the VoiceVox Docker image or on Google Colab, and that the Colab route requires changing voicevox_url in utils/TTS.py to the link Colab gives you.

voicevox_url = 'http://localhost:50021'

Finally, pick the TTS engine by commenting and uncommenting in run.py. The README shows voicevox_tts(tts) active and the Silero call commented, with the note that Silero input must be in the language you request.

# Choose between the available TTS engines
voicevox_tts(tts)

# silero_tts(tts_en, "en", "v3_en", "en_21")

What you should see after a run is a transcript printed as "Question: ..." and an output.wav containing the reply. If you hear nothing, the README's own troubleshooting advice is to listen to input.wav to confirm your microphone was actually captured.

The openai==0.28.1 pin is the real constraint

Most projects treat a dependency pin as a footnote. Here it is the central engineering decision. The FAQ answers the transcribing error by telling you not to upgrade the OpenAI library, and requirements.txt carries the same instruction in a comment. The code depends on the pre-1.0 module layout, so any attempt to modernise the OpenAI client means rewriting the transcription and chat calls, not bumping a version string.

A second trap sits in the same function. The shipped transcribe_audio wraps the API call in a bare try/except that prints "Error transcribing audio" and returns, which hides the actual exception. The README's fix is to delete the try/except so the real error surfaces. That is good advice and also an admission that the default error path is unhelpful.

There is also a Mecab warning. The README describes mecab-python3 as tricky to install and says that if it fails you can delete the katakana_converter call in utils/TTS.py, since that function is optional. So a Japanese-specific text normalisation step is treated as removable rather than required.

VoiceVox versus Silero, and why the choice is not only about voice quality

The two TTS backends impose different language contracts. VoiceVox accepts Japanese input only, which is why the README insists you translate the model's answer to Japanese first. Silero, added in v3.0 according to the update notes, covers Russian, English, German, Spanish, French, Tatar, Ukrainian, Uzbek, Kalmyk and Hindi, and the README warns that the input must be in the target language. Voice samples for Silero are linked from the project's own samples page, and VoiceVox speaker ids live in speaker.json and are changed in utils/TTS.py.

The practical split is this: VoiceVox gives you an anime-styled Japanese voice with a speaker catalogue, at the cost of a running engine and a mandatory translation step. Silero gives you more languages and can skip translation entirely if your persona already answers in that language, at the cost of a different voice character. Neither is described in the README in terms of speed or resource use, so that comparison has to be made on your own machine.

Where AI-Waifu-Vtuber breaks, and when it is the wrong tool

The failure modes are mostly environmental. VTube Studio does not receive audio from this program directly; the README's step 8 is a manual routing job through VB-Cable, and if the wrong device is selected the avatar simply stays still while the program appears to work. Twitch mode needs an OAuth token from the tmi endpoint pasted into utils/twitch_config.py, along with the server, port, nickname, user and channel fields, and the README notes the nickname does not need changing.

Translation is a soft dependency with a hard consequence. Google Translate is the simple option, DeepLx is the free DeepL-compatible route that needs no API key and can run in Docker, and the README's author states plainly that DeepL output is more accurate and casual while explaining that DeepL registration was unavailable in their country. If you skip translation on the VoiceVox path, you are feeding non-Japanese text to an engine that expects Japanese.

This is the wrong tool if you want a maintained product. The last push to the repository was on 2026-05-31, there are no retrieved releases, and the licence field is unknown. There is no published licence file in the top-level listing, which matters if you intend to reuse the code or ship a modified version. It is also the wrong tool if you need a supported OpenAI SDK, because the pinned 0.28.1 is the whole reason the code runs at all.

Alternatives and the difference in approach

The README names shioridotdev as the inspiration, and the honest comparison is architectural rather than feature-by-feature. AI-Waifu-Vtuber is a monolithic script: one process, global conversation state, file-based audio handoff, and configuration by editing Python. An alternative built around the current OpenAI Python SDK would separate the client, use a maintained audio API surface, and avoid the httpx conflict that forces googletrans==4.0.0rc1 to stay where it is.

On the voice side, the alternative to VoiceVox is not another cloud service but the engine choice already in the repository: Silero removes the translation hop for non-Japanese personas. If your persona speaks English, that single change eliminates an entire stage of the pipeline and the failure surface that comes with it. That is a more meaningful difference than swapping in a different LLM.

Maintenance, upgrade cost and licence status

The last push was on 2026-05-31, so the project is not archived but it is also not being changed frequently. There are no retrieved releases, which means upgrades arrive as commits on master rather than versioned artefacts. The upgrade cost is concentrated in one place: openai==0.28.1. Any future move to the 1.x client is a code migration across the transcription and chat calls, and it also reopens the httpx conflict with googletrans.

On licensing, the repository's licence is unknown and no licence file appears in the top-level entries. That is a gap you should resolve with the repository owner before redistributing anything, and it is not something I can resolve for you here. The README does credit the upstream projects it depends on, including VoiceVox, Silero, DeepL and Whisper, each of which carries its own terms.

Editorial conclusion

Adopt AI-Waifu-Vtuber if you already run VTube Studio, can keep openai==0.28.1 pinned, and want a single run.py to experiment with a talking avatar rather than a maintained platform. Skip it if you need a supported SDK, a documented licence, or a project with a release history, because none of those are in the repository. Verify first that Whisper transcription works with your microphone and that your chosen TTS engine answers on its configured URL, since both failures are silent in the default try/except.

Frequently asked questions

Official sources

1. ardha27/AI-Waifu-Vtuber on GitHub
2. Issues
3. README