ZerolanLiveRobot: a Python control framework for a streaming AI VTuber
AI VTuber with LLM, ASR, TTS, OCR, CV and more technologies to live stream or play Minecraft with you.
At a glance
- What is it?
- ZerolanLiveRobot wires ASR, an LLM, TTS, screen OCR, image captioning, Live2D and a Minecraft agent into one livestream bot. It is a control framework, not a model bundle, so the AI services it calls have to exist somewhere else first.
- Who is it for?
- Adopt ZerolanLiveRobot if you already run or plan to run ZerolanCore, accept HTTP services for ASR, LLM and TTS, and want the orchestration, Live2D window, OBS subtitle wiring and Minecraft agent handled for you. Skip it if you want a single pip install that contains working models, or if you only stream on YouTube or Twitch, since the README marks both as experimental while Bilibili is listed without that caveat.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 155 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap ZerolanLiveRobot fills between models and a stream
A working AI VTuber needs several models that do not know about each other. Speech has to become text, text has to become a reply, the reply has to become audio, and something has to decide when to speak, what to look at, and which tool to call. ZerolanLiveRobot is that something. The README describes it as the control framework of the livestream robot, one that collects data from several sources and synthesizes an action response. The models live in a sibling repository, ZerolanCore, which the README calls the AI driving core and which exposes speech recognition, large language model and text to speech as web API services.
The intended audience is narrow on purpose. The README addresses people who watched Neuro-sama or the Chinese equivalent 木几萌 and want their own avatar for streaming, chatting and gaming, and it claims this needs only a consumer-grade graphics card. That claim is about the hardware the models run on, not about this repository, which is Python glue. If you are building a research pipeline for speech or vision, this project is the wrong layer. If you want an agent that reads danmaku, watches a window, talks back with emotion and can be told to control a Minecraft character, the scope matches.
Five repositories, one running system
The project is deliberately split. ZerolanLiveRobot is the control framework. ZerolanCore serves inference. ZerolanData defines the data formats exchanged between projects and services. ZerolanPlayground is a Unity and Vuforia AR avatar viewer that also displays Live2D models. KonekoMinecraftBot is a mineflayer-based Minecraft agent whose behaviour is governed by a finite state machine (fighting, chopping trees, sleeping) and which accepts voice control.
The interfaces between them are HTTP. The README points at the pipeline module inside this repository and at the interfaces defined in ZerolanData, and states plainly that they are all implemented as HTTP interfaces. That single sentence explains most of the deployment consequences later in this article. There is no in-process model loading here, no shared GPU memory pool, no Python object handoff. Every recognition, generation and synthesis step is a network round trip to an endpoint you configure.
The README also names a Unified Model Pipeline, or UMP, as the extension point. If none of the supported third-party APIs suits you, you implement your own pipeline against those interfaces. Mixing is explicitly allowed: a third-party LLM alongside self-hosted ASR and TTS, selected in the configuration file.
From f8 keypress to spoken reply
The documented interaction loop is short and worth reading closely, because it tells you what is and is not automatic. By default, f8 toggles the microphone. You press it once, speak, and press it again. The captured audio goes to the ASR service. When ASR returns text, that text is passed to the LLM service. The LLM response is handed to TTS for synthesis, and the README says you should then hear the robot's reply.
Everything else in the feature list hangs off that spine. Danmaku from Bilibili, YouTube or Twitch is read and the bot picks which messages to answer. Screen text in a chosen window is read by OCR and analyzed, and images from that window are captioned. A browser can be opened and driven for encyclopedia searches, and voice commands can move the mouse to click UI elements. Memory is two-tier: a short-term runtime context capped by a maximum record count, and a long-term store backed by a vector database, which the README illustrates with a question about whether the bot remembers what 春日影 is.
Output side, the Live2D viewer is built on live2d-py, OpenGL and PyQt5, and handles window sizing, transparent backgrounds for OBS or desktop pet use, automatic breathing and blinking, and mouth movement while speaking. A Unity-based viewer exists for Live2D and 3D models, and QQ bot replies support text, voice and images.
Install commands and the config file the first run writes
The README requires git up front, for cloning, updating and installing ZerolanData. On Windows it also asks you to check that Visual C++ Build Tools are present, since some Python dependencies may need compilation. The documented setup uses conda with Python 3.11, which the README recommends, and the project badge states Python 3.10 to 3.11:
conda create --name ZerolanLiveRobot python=3.11 conda activate ZerolanLiveRobot pip install -r requirements.txt
Running python main.py the first time does not start the bot. It generates ./resources/config.yaml and exits, which the README calls normal. You then edit that file in one of two ways. python webui.py launches a browser configuration interface, usually at http://127.0.0.1:7860, where you fill in fields and click Save Config in the top right, which writes back to ./resources/config.yaml. Or you edit the YAML directly, following the comments inside it. The README recommends reading those comments, and having seen how much of the runtime behaviour is described only there, that is fair advice.
Three services are mandatory before anything works: an LLM, an ASR model and a TTS model. The README states this as a hard floor. OBS integration is optional but concrete. Enable the WebSocket server in OBS under Tools, WebSocket Server Settings, generate a password, show connection info, and put the IP, port and password into the config. Then add two Text (GDI+) sources and rename them exactly UserText and AssistantText, the first showing recognized user speech and the second showing the model's text output. Updates come from main: git checkout main followed by git pull, with the README warning that local edits may auto-merge and that unresolved conflicts are yours to handle.
The models are not in this repository
This is the limitation that decides most adoption questions. Installing ZerolanLiveRobot gives you orchestration, a Live2D window, OBS subtitle control and a voice command surface. It does not give you a speech recognizer, a language model or a synthesizer. The README sends you to ZerolanCore to deploy those yourself, or to third-party APIs, and repeats that at minimum LLM, ASR and TTS must be configured to drive the whole project.
A second constraint follows from the HTTP design. Because every step is a service call, latency compounds: your audio waits for ASR, the transcript waits for the LLM, and the reply waits for TTS before anything is audible. The README gives no latency figures and no throughput numbers, so treat the responsiveness of the finished bot as something you measure on your own hardware rather than something the project promises. The consumer-grade GPU claim refers to running the models, and it says nothing about how many of them fit on one card at once.
Platform support is uneven. Bilibili is listed as supported, while YouTube and Twitch carry an experimental label in the README itself. The QQ bot feature and the AR viewer via ZerolanPlayground are additional surfaces, with the AR path marked optional because Live2D already covers avatar display. If your audience is not on Bilibili, you are working against the project's best-tested path.
ZerolanLiveRobot against a single-repository VTuber stack
The obvious alternative is a monolithic AI VTuber project: one repository that bundles the models and the streaming logic, installed with one command. The difference is not quality, it is where the boundaries sit. A bundled project hides the model choice from you and upgrades everything together. ZerolanLiveRobot exposes each stage as an HTTP endpoint behind the UMP interfaces in the pipeline module, so you can point ASR at one machine and the LLM at a paid API, and swap either without touching the control logic.
That trade costs you a multi-repository install. You clone this project, install ZerolanData through git, and stand up ZerolanCore separately if you are self-hosting. You also inherit version coupling across five repositories that share data formats through ZerolanData. The README does not describe a compatibility matrix between them, so when you update one, the others are your responsibility. For a solo streamer who wants the shortest path to a talking avatar, a bundled stack is less work. For someone who already has inference services, or who intends to change models as better ones appear, the HTTP seam is the reason to pick this one.
Licence, upgrade path and what maintenance actually costs
The project is MIT licensed, which permits commercial and private use with the usual requirement to keep the copyright and permission notice. That covers this repository's code. It does not automatically cover the AI models you connect, the Live2D runtime, or third-party APIs you configure, each of which carries its own terms. Nothing here is legal advice, and if you plan to monetize a stream built on this stack, the model and Live2D licences are the ones to read, not this one.
Upgrades are branch-based rather than release-based in practice. Releases exist, with v2.3.0 in January 2026 following v2.2.0 and v2.1.4 earlier that month, but the README's update instructions pull main. There is no documented migration procedure for config.yaml between versions, and since the first run generates that file and later runs read it, a schema change would surface as a startup error rather than a guided migration. Keeping your configuration in version control and diffing it after each pull is the cheapest safeguard, and the README's own suggestion to develop on a branch other than main applies here too.
Ongoing cost is dominated by what you connect, not by this repository. Self-hosted ASR, LLM and TTS consume GPU time whenever the bot is live; API-based services bill per request. The Python side is small enough that the real maintenance surface is the set of endpoints in config.yaml and the five-repository version alignment behind them.
Editorial conclusion
Adopt ZerolanLiveRobot if you already run or plan to run ZerolanCore, accept HTTP services for ASR, LLM and TTS, and want the orchestration, Live2D window, OBS subtitle wiring and Minecraft agent handled for you. Skip it if you want a single pip install that contains working models, or if you only stream on YouTube or Twitch, since the README marks both as experimental while Bilibili is listed without that caveat. Verify first that Python 3.11 plus the Visual C++ Build Tools path builds your dependencies on Windows, and that your chosen LLM, ASR and TTS endpoints answer before you touch the rest of config.yaml.
Community notes