ChatdollKit: Turning a Unity 3D Model into a Voice Chatbot

What ChatdollKit solves, and who it is actually for

Building a talking character is mostly plumbing. You need a speech recognizer, a turn-end detector, an LLM call, a text-to-speech provider, and then something that moves the model's mouth and body in time with the audio. ChatdollKit's README describes it as a "3D virtual assistant SDK that enables you to make your 3D model into a voice-enabled chatbot", and the feature list confirms that the SDK covers that whole chain rather than one link in it: generative AI providers (ChatGPT, Anthropic Claude, Google Gemini Pro, Dify), speech-to-text and text-to-speech (OpenAI, Azure, Google, VOICEVOX / AivisSpeech, Aivis Cloud API, Style-Bert-VITS2), dialog state management, intent extraction, topic routing and wakeword detection.

The intended user is a Unity developer with a VRM model and an idea for a character. The topics on the repository include vrm, unity3d and ai-companion, and the README points to a shipped iOS app, OshaberiAI, as an example of what the SDK produces. If you are not in Unity, the SDK has little to offer you: the primary language is C# and the top-level layout is a Unity package structure (Scripts/, Prefabs/, Plugins/, Editor/, Textures/, Audio/).

The pipeline: LLM, STT, TTS and a model controller

The README ships an architecture diagram at Documents/Images/chatdollkit_architecture_overview.png, and the 0.8.16 release notes describe a refactor that clarifies the split: speech handling was extracted into a component called SpeechController and face expressions into FaceController. That is the shape of the runtime. A listener produces text, the dialog component decides what to say next, a synthesizer produces audio, and the controllers drive the model.

Two recent changes are worth understanding because they affect latency. Version 0.8.16 added WebSocket streaming speech recognition that, in the release notes' wording, "offloads VAD to the server and completes recognition during turn-end detection, reducing overall response latency by several hundred milliseconds." The same release added barge-in, so the user can interrupt the character mid-sentence. Earlier, 0.8.13 added Silero VAD, and 0.8.15 described combining "multiple voice activity detection methods (e.g., Silero VAD + built-in energy-based VAD)" for noisy venues. Turn-end detection is clearly the part of the pipeline the maintainer has spent the most effort on, which matches where voice assistants usually fail.

State is handled in-process. The README lists dialog state (context) management as a built-in feature, and 0.8.10 added long-term memory with components for ChatMemory, plus the option to integrate services like mem0 or Zep. So there are two tiers: short context inside the SDK, and searchable history if you wire up a memory backend.

Getting a first conversation running from the repository

The README does not give a package-manager command or an install step. What it does give is the repository layout plus a set of Unity scenes under Demo/, including Demo/Demo08.unity and a Demo/Modular/ folder, and the 0.8.15 notes say the UI controls work "out-of-the-box with zero configuration, just drop them onto your scene's Canvas." The practical first step is therefore to get the repository and open it as a Unity project, then open the demo scene.

The README links a WebGL live demo, so you can see the intended behaviour before building anything locally:

https://unagiken.blob.core.windows.net/chatdollkit/ChatdollKitDemoWebGL/index.html

That URL is the demo the README points at. The README says to say "Hello" to start the conversation, and that the character is multilingual, so you can ask her to switch languages.

Once you move to a local project, the repository itself is the thing you open. The top-level entries are .gitattributes, .gitignore, Audio/, Demo/, Documents/, Editor/, Examples/, Extension/, LICENSE, Plugins/, Prefabs/, README.ja.md, README.md, Scripts/, Tests/ and Textures/. Open the project in Unity and load the demo scene that matches your target platform. Note that the SDK is released under Apache-2.0, so check the LICENSE file at the repository root for the exact terms before you redistribute anything.

When you move to your own scene, the pattern the README describes for UI is to add the provided controls to the Canvas. For the model, 0.8.6 notes that VRM runtime loading was improved to allow switching 3D models at runtime, so you are not locked to the demo avatar. The SDK also exposes components for the individual pieces (listeners, synthesizers, dialog) which the Demo/Modular/ folder is presumably there to illustrate; the README points at the demos for details rather than documenting each component inline, which is a gap you will notice once you go past the sample scenes.

Where ChatdollKit is the wrong choice

The obvious limitation is the platform. ChatdollKit is a Unity SDK with a C# codebase and a Unity folder layout. If your chatbot is a web service, a Slack bot, or a backend agent that never renders a character, this is the wrong layer. The README's platform list (Windows, Mac, Linux, iOS, Android, VR, AR, WebGL) is about where Unity builds run, not about where the SDK can be embedded independently of Unity.

A second limitation is provider churn. The release notes read as a moving target: 0.8.13 removed OpenAI-specific parameters from the OpenAI-style endpoint so that Grok and Gemini work, 0.8.14 added Aivis Cloud API and AIAvatarKit TTS/STT plus a GPT-5 reasoning_effort parameter, 0.8.8 and 0.8.9 added NijiVoice. Each of those is a new integration to keep working. If you pin a version, you keep the integrations that existed then; if you track master, you inherit the changes. The README does not document a deprecation policy or a rollback procedure for provider integrations, so plan to read release notes before upgrading.

Third, the documentation is uneven. The README is a feature list and a changelog, not a reference manual. It defers to the demos for component-level detail, and there is a separate README.ja.md for Japanese readers. Expect to read source under Scripts/ and Extension/ for anything the demos do not cover. Finally, 0.8.4 removed legacy components and pointed readers to a migration section for 0.7.x, which tells you the API has broken between minor versions before and may again.

ChatdollKit versus building the stack yourself or using a server-side agent framework

The real alternative is not another Unity SDK; it is assembling the pieces yourself. You would pick a speech recognizer, write your own turn-end logic, call an LLM, call a TTS API, and write the animation timing code that makes the mouth move with the audio. That gives you full control and no Unity dependency, but you own every integration and every latency regression.

A second alternative is the server-side agent route, and ChatdollKit itself acknowledges it. Version 0.8.11 and 0.8.12 added an AIAvatarKit backend that "offloads AI agent logic to the server", letting you plug in frameworks like AutoGen for capability expansion. That is the opposite trade-off: the agent logic moves off the client, the Unity app becomes a thin renderer, and you gain access to agent SDKs at the cost of running and securing a backend. If your character needs tools, multi-step reasoning or shared state across users, the server-side split is the more sensible architecture, and the SDK supports it rather than competing with it.

For pure animation and lip-sync without the LLM stack, a general Unity VRM toolkit would be lighter. ChatdollKit's value is precisely that it bundles the conversation loop, so choosing it means accepting the bundle.

Maintenance, releases and what upgrading costs you

The repository is not archived, and the last push was on 2026-09-10, four days before this writing, so the project is being worked on. The release cadence is uneven rather than regular: v0.8.14 on 2025-08-12, v0.8.15 on 2025-08-21, then v0.8.16 on 2026-02-14. The gap between 0.8.15 and 0.8.16 is roughly six months, and the 0.8.16 notes describe substantial work in that window (streaming STT, barge-in, the SpeechController and FaceController refactor), so a quiet release channel does not mean a quiet codebase.

Upgrade cost concentrates in two places. First, renamed or extracted components: the 0.8.16 refactor moved speech handling and face expressions into new classes, so code that reached into the old model controller will need adjusting. Second, provider configuration, since the release notes show parameters being added and removed at the endpoint level. Neither is documented as a formal migration path in the README beyond the 0.7.x migration section.

On licensing, the repository is Apache-2.0. That permits commercial use and modification with the usual conditions around notices and attribution, but it says nothing about the third-party services you connect to. Your OpenAI, Azure, Google, VOICEVOX or Aivis Cloud usage is governed by those providers' terms, and the SDK does not mediate that. This is not legal advice; read LICENSE and each provider's terms yourself.

Editorial conclusion

Adopt ChatdollKit if you are already working in Unity, have a VRM model you want to animate, and would rather assemble LLM, STT and TTS providers from a documented set of components than write the orchestration yourself. Do not adopt it if you want a headless server-side agent, or if your project cannot take a Unity dependency. Before committing, open the Demo scene that matches your target platform and confirm the providers you plan to use are actually present in this version, because the release notes show the supported provider list changing between minor releases. Start by checking that Demo08.unity builds for your platform.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. uezo/ChatdollKit on GitHub