Model or dataset
uezo/ChatdollKit avatar
uezo/ChatdollKit

ChatdollKit's hard part is turn-taking, not lip sync

ChatdollKit enables you to make your 3D model into a chatbot

1,283 stars129 forksC#Apache-2.0

At a glance

What is it?
A Unity SDK that wires an LLM conversation onto a 3D or VRM model, where most of the release history since 0.8.10 has gone into voice activity detection, streaming recognition and interruption handling rather than into the animation layer.
Who is it for?
ChatdollKit fits a Unity team that already has a VRM model and wants a talking character with interruption handling, since the hard parts of turn-taking are handled rather than left to the caller. It does not fit a web project that has no Unity runtime, and it does not save you from model hosting, since both speech services and the language backend are external.
Can I use it commercially?
Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 15 days ago.
What is it written in?
Mainly C#, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • uezo/ChatdollKit1,283 ★ · +62
Star history
Repository2026-09-142026-11-15
uezo/ChatdollKit1.2k1.3k

62 Stars in the last 62 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

Recognition moved to a WebSocket stream in 0.8.16, and VAD left the device

The 0.8.16 release, tagged on 2026-02-14, changed how audio reaches the recognizer. Streaming speech recognition now runs over a WebSocket, voice activity detection is offloaded to the server, and recognition completes during turn-end detection instead of after it. The stated effect is a cut of several hundred milliseconds from end to end. That is a server round trip moved earlier in the conversation rather than a cheaper model, and it only pays off if the endpoint you point at supports streaming. The earlier approach is still in the tree: AzureStreamSpeechListener arrived back in 0.8.3 to recognize speech as it is spoken, which was a streaming path for one provider. So the WebSocket work is the general case of something that was previously Azure specific, and a project already wired to that listener has a second option to evaluate rather than a replacement to make.

Barge-in needed the volume ducking to work, which came a release earlier

Interrupting the character mid-sentence is the feature 0.8.16 calls barge-in, and on its own it is a small change. The supporting work landed in 0.8.14, where automatic volume control was added for the case where a user interrupts while the character is still speaking, and the same release claimed a fix for conversation breakdown caused by turn-end misrecognition. That pairing is the actual design: an interruption the microphone detects has to be audible over the character that is currently talking, and it has to be distinguishable from the tail of the character's own speech leaking into the microphone. Echo cancelling is part of the same problem, and 0.8.14 added native microphone support on Android, iOS, and macOSX for devices that provide acoustic echo cancelling and noise cancelling natively. Turning is an older feature than it looks: 0.8.3 already allowed interrupting a character to take your turn, and paired it with natural pauses between exchanges. What 0.8.16 added is interrupting mid-sentence rather than at a boundary, which is a harder target for the detector because the tail of the model's own audio is still in the buffer. The WebGL build has no native echo cancelling path, which is part of why its noise handling is a separate topic.

Two voice activity detectors run together because one of them misreads turn ends

The turn-end detector is where most of the iteration has gone. Version 0.8.13 added Silero VAD, described as machine learning based and much more accurate than an energy threshold, with outdoor and event use as the target. Version 0.8.15 went further and combined Silero VAD with the built-in energy-based detector rather than replacing it, so a detection has to be agreed by both. That is a deliberate redundancy: the energy detector is cheap and never misses a loud syllable, while the learned detector is the one that stops early on a trailing vowel. The same release brought Silero VAD to WebGL builds. An integrator should expect to tune thresholds here. Nothing in the release notes states a default sensitivity, so the practical starting point is the combination the project ships and adjust from observed behaviour in a noisy room.

The OpenAI-style endpoint stopped sending OpenAI-only parameters in 0.8.13

Model portability came from deleting things. In 0.8.13 the OpenAI-specific parameters were removed from the OpenAI-style endpoint, which is what lets Grok, Gemini, and other API-compatible models work without code changes. Before that, any provider that did not recognize those fields had to be special-cased. The named backends are ChatGPT, Anthropic Claude, Google Gemini Pro, and Dify, with function calling supported for ChatGPT and Gemini and multimodal capability listed alongside. Version 0.8.14 extended the same endpoint with the GPT-5 reasoning_effort parameter, and 0.8.14 also added Aivis Cloud API for speech synthesis and AIAvatarKit for both speech recognition and synthesis. The practical consequence is that swapping language backends is a configuration change, while swapping speech services is closer to a component swap.

The agent loop can be pushed onto a server through AIAvatarKit

Since 0.8.11 and 0.8.12, the toolkit will hand its agent logic to a backend called AIAvatarKit, described as offloading AI agent logic to the server to keep the front end maintainable. The point is that the client no longer has to own the conversation loop. Any agent SDK can be plugged into that backend, with AutoGen named as the example, so the ceiling on character behaviour is set by what the server runs rather than by what the Unity scene can express. Earlier releases had already pushed flexibility outward in two directions: 0.8.8 and 0.8.9 added Dify as a backend and let AITuber characters talk to each other, which is what turns a single character into a scene of several. Long-term memory arrived in the same 0.8.10 release that made the language switching automatic.

Long-term memory is a component you swap in, not a store the kit keeps

Version 0.8.10 added the ability to store past conversation history and search it, and the implementation detail matters more than the headline. The components come from a separate project, ChatMemory, and the release explicitly leaves room to integrate other services such as mem0 or Zep instead. In other words the SDK defines the interface and not the database, and a project that already keeps its own log has a path that avoids running a second copy. That same 0.8.10 release made multi-language handling automatic in both directions, so the character can switch the language it speaks and the language it listens for during a conversation rather than being pinned at startup. The WebGL demo shows the other half of that: saying hello starts the conversation, and asking to talk in Japanese switches it.

WebGL builds run on AudioWorkletNode, and API keys sit in the browser

The browser build carries constraints that the native targets do not. Microphone capture was moved to AudioWorkletNode for lower latency and steadier behaviour, mute and unmute handling was stabilised, and error handling was changed so HTTP errors surface immediately instead of leaving the request hanging. The same release fixed API-key authorization in WebGL builds, which is the cost of running the model call from a browser tab: the credential ships to the client whether or not that is wanted. 0.8.15 added camera switching between front and rear with correct aspect ratio handling, image file upload, and a fix for lip sync when muted. The live demo is a WebGL build, so it exercises this path rather than the editor path, which is useful for seeing failure modes and useless for measuring desktop behaviour.

The repository is a Unity project, not a package with an install step

Everything in the tree assumes a Unity project rather than a library boundary. The top level holds Audio/, Demo/, Documents/, Editor/, Examples/, Extension/, Plugins/, Prefabs/, Scripts/, Tests/, and Textures/, plus a Japanese README alongside the English one, and the default branch is master rather than main. The Demo folder is the working reference: Demo08.unity is the current scene, and it carries controller assets for two modes, AGIA.controller and AGIAFree.controller, alongside AITuber/, Model/, Modular/, Scripts/, and Textures/ subfolders. That pairing of scenes and controllers is the pattern a new project should copy, since the release notes for 0.8.3 describe easier animation registration as the change that cleaned up the code, and 0.8.6 improved VRM model loading at runtime so models can be swapped without an error. Two earlier structural changes are worth knowing before starting. Version 0.8.4 reorganized the key components around modularity and deleted the outdated ones, which is the release that carries a migration path from the 0.7.x line; anyone arriving from 0.7 has to follow that path rather than merge the files. Version 0.8.2 added the ability to drive the Unity application from JavaScript in a WebGL build and introduced the SpeechSynthesizer component, which is the hook that later TTS providers plug into. Licensing is Apache-2.0.

Editorial conclusion

ChatdollKit fits a Unity team that already has a VRM model and wants a talking character with interruption handling, since the hard parts of turn-taking are handled rather than left to the caller. It does not fit a web project that has no Unity runtime, and it does not save you from model hosting, since both speech services and the language backend are external. Before committing, check which speech service you will pay for, confirm the release you target matches your Unity version, and read the migration notes if you are moving off the 0.7.x line.

Frequently asked questions

Which speech services can ChatdollKit use for recognition and synthesis?

Speech-to-text and text-to-speech are both pluggable, with OpenAI, Azure, Google, VOICEVOX and AivisSpeech, Aivis Cloud API, and Style-Bert-VITS2 named. NijiVoice was added as a synthesizer in 0.8.8, and 0.8.14 added AIAvatarKit for both directions.

How does ChatdollKit decide that the user has finished speaking?

Version 0.8.13 added Silero VAD, and 0.8.15 combines it with the built-in energy-based detector so both have to agree on a turn end. The 0.8.16 WebSocket path moves voice activity detection to the server and finishes recognition during turn-end detection.

Can ChatdollKit drive a language model other than ChatGPT?

Yes. ChatGPT, Anthropic Claude, Google Gemini Pro, and Dify are named, and 0.8.13 removed the OpenAI-specific parameters from the OpenAI-style endpoint so Grok, Gemini, and other compatible models work unchanged. 0.8.14 added the GPT-5 reasoning_effort parameter.

Does ChatdollKit remember earlier conversations?

Since 0.8.10 past conversation history can be stored and searched. The components come from the separate ChatMemory project, and the release also points at mem0 or Zep as services to integrate instead.

Which platforms does a ChatdollKit build run on?

Windows, Mac, Linux, iOS, Android, and other Unity-supported platforms including VR, AR, and WebGL. WebGL builds capture audio through AudioWorkletNode and can be driven from JavaScript, as added in 0.8.2.

Official sources

  1. Issues
  2. License: Apache-2.0
  3. README
  4. Releases
  5. uezo/ChatdollKit on GitHub
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/uezo-chatdollkit.svg)](https://hysenlabs.com/projects/uezo-chatdollkit)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/uezo-chatdollkit.svg)](https://hysenlabs.com/projects/uezo-chatdollkit)

Was this useful?