AIAvatarKit: a Python speech-to-speech pipeline for conversational avatars

What AIAvatarKit actually solves for avatar builders

The README frames the project around one job: turning what a user says into what an avatar says and does. The listed targets are talking avatars inside web or mobile apps, interactive signage and virtual store staff, companion hardware such as Raspberry Pi, M5Stack and StackChan, metaverse characters on VRChat, cluster and Vket Cloud, phone operators through Twilio or Asterisk, and multi-channel assistants. That list is broad, but it describes one shape of problem. You have a channel where a human speaks, and you need a low-latency loop that transcribes, reasons, speaks back, and can also drive a face, an animation or an on-screen artifact.

The audience is therefore Python developers who are willing to assemble their own stack. The project does not hide the components behind a single vendor. VAD, STT, LLM and TTS are swappable modules, and the README states that a small interface covers providers beyond the built-in ones. If your goal is a chatbot that only exchanges text, the speech pipeline is overhead. If your goal is a voice agent where you control which model answers and which voice speaks, the modular layout is the point.

The STSPipeline and the adapter boundary

The architecture section describes a single STSPipeline that streams each stage into the next: VAD, then STT, then the LLM with tools and agent logic, then TTS. A branch from the LLM also produces face, animation and artifact output. The README does not claim that every stage is fully parallel in all configurations, but it does state that streaming and parallelism run throughout the pipeline, including speculative STT and a spoken filler before the answer itself.

Around that pipeline sit adapters. An adapter wraps the pipeline for one channel (WebSocket, HTTP, telephony, messaging) and owns only transport concerns. The design consequence is stated directly: multiple adapters can attach to the same pipeline instance, so a user can move between channels within one conversation. That is the mechanism behind the omnichannel claim. It also means conversation state lives with the pipeline, not with the socket, which is the part worth checking against your own session model. Turn-end detection is its own layer. The README lists turn-end gates separately from VAD: Smart Turn, Namo Turn, filler-only, LLM-based, session hold, and custom. The built-in application uses Namo Turn semantic VAD by default.

Installing AIAvatarKit and running the built-in app

The README states the requirements plainly: Python 3.11+, an OpenAI API key, and a reachable VOICEVOX-compatible server at its default URL. The package installs from PyPI.

pip install aiavatar

The built-in application starts from the command line once the key is exported. The README gives exactly this pair of commands.

export OPENAI_API_KEY=sk-xxx
aiavatar

On first start, the built-in application uses Namo Turn semantic VAD. If its optional dependencies are missing, the CLI asks whether to install them, with the prompt shown in the README. Answering y installs aiavatar[namo-turn] into the current Python environment and continues startup. Answering n starts without the Namo Turn gate, leaving Silero VAD and the filler gate enabled. Once running, the avatar UI is at http://127.0.0.1:8000/ and the Admin Panel at http://127.0.0.1:8000/admin/. The README notes that VOICEVOX must be launched beforehand.

For full control over components, routes and application lifecycle, the README provides a script mode. It builds an AIAvatarWebSocketServer, mounts the WebSocket router, optionally attaches the admin panel, and serves a downloaded example UI at the root.

import os

from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from aiavatar.adapter.websocket.server import AIAvatarWebSocketServer
from aiavatar.admin import setup_admin_panel
from aiavatar.util import download_example

html_dir = download_example("websocket/html")

aiavatar_app = AIAvatarWebSocketServer(
    openai_api_key=os.environ["OPENAI_API_KEY"]
)

app = FastAPI()
app.include_router(aiavatar_app.get_websocket_router())

The README warns that the catch-all mount for the UI must come after the routes above, and that the server starts with python -m uvicorn run:app, again with VOICEVOX already running. The .env.example file documents a second configuration path: copy it to .env, set OPENAI_API_KEY, and run aiavatar. It states that the command loads .env from the current working directory without replacing variables already present in the process environment, and that command-line options override their corresponding environment variables.

Where AIAvatarKit gets in your way

The default path assumes infrastructure you may not want. The built-in application expects a VOICEVOX-compatible server at its default URL, and the README says to launch VOICEVOX beforehand. That is a local service dependency sitting next to your Python process. If your deployment target cannot run it, you are not on the default path anymore, and the README points to documents/getting-started.md for the full set of settings rather than promising a hosted fallback.

The optional semantic VAD dependencies are a second friction point. Namo Turn and Smart Turn both pull onnxruntime, transformers and huggingface-hub. The interactive prompt handles this for the CLI, but a non-interactive startup has no one to answer it. The .env.example is explicit that when OPENAI_API_KEY is absent, an interactive CLI prompts securely while non-interactive startup fails. Both behaviours point the same way: this is a project you configure deliberately, not one that boots unattended without setup.

Version drift is a third issue the README raises on its own. It warns that technical blog posts may be based on a version prior to v0.6, and suggests pip install aiavatar==0.5.8 to match such an environment. That is a useful admission, and it also means older tutorials circulating online describe a different API surface than the current release line, which is at v0.9.0.

AIAvatarKit compared with a hosted voice-agent platform

The honest alternative is a hosted voice-agent service, where you configure a prompt, pick a voice, and the vendor runs the audio path. The difference is not quality, it is where the seams are. With a hosted platform you get one provider's VAD, STT, LLM and TTS, tuned together, and no local services. With AIAvatarKit you assemble the pipeline yourself from the shipped implementations, which the README lists by component: Silero VAD, Azure Speech, Amazon Transcribe and Parapper for detection; Azure, Google Cloud, OpenAI and AmiVoice for speech-to-text; OpenAI Chat Completions, Azure OpenAI, the OpenAI Responses API, Anthropic Claude, Google Gemini, xAI Grok and OpenRouter for the model.

That table is the real argument for the project. You can change the voice without touching the transport layer, or change the model without touching the UI. The cost is that you own the deployment: the VOICEVOX dependency, the optional ONNX packages, the API keys, and the process lifecycle. A hosted platform also gives you one channel by default. AIAvatarKit's adapter model is built so that one pipeline instance serves several channels and the conversation follows the user, which the README illustrates with hanging up a phone call and continuing on LINE.

Agent tools, guardrails and the admin surface

The README states that tool calls and MCP are supported, and describes two behaviours that matter for latency. Tools load only when needed, so a large catalog does not confuse the model, and slow tools either run in the background or return a reply straight from a template instead of stalling the conversation. Those are design claims from the documentation, not measured results, but they describe a specific mechanism rather than a general promise.

Guardrails are described as running in parallel, with the ability to interrupt the avatar mid-sentence to correct what it just said. That is an unusual capability and worth verifying against your own use case, because interrupting speech implies the guardrail result can arrive after the TTS stage has already begun. The Admin Panel is listed as covering config, logs, metrics and evaluation, plus Langfuse tracing, and the README says a running pipeline can be retuned without restarting. For a system where you are constantly swapping models, being able to change settings on a live process is a practical advantage over editing a script and restarting.

Maintenance, licensing and what to pin

The repository is not archived, and the last push was on 2026-09-10. The most recent release listed is v0.9.0 from 2026-08-29, following v0.8.19 and v0.8.18. That cadence suggests the project is moving, and the README's own note about pre-v0.6 blog posts is a reminder that the API has changed across versions. If you build on it, pin the version you validated rather than tracking the package loosely.

The licence is Apache-2.0, declared in pyproject.toml as license = { text = "Apache-2.0" } and shipped as a LICENSE file at the repository root. That is a permissive licence, but it covers this project's code only. The services you connect (OpenAI, Azure, Google, Anthropic, and so on) carry their own terms, and VOICEVOX is a separate component with its own licensing that the README does not discuss. The project does not offer legal guidance, and neither does this article. Check the terms of each provider and of VOICEVOX itself before shipping.

The dependency list is modest but not trivial: httpx, openai, aiofiles, numpy, silero-vad, onnxruntime, websockets, fastapi, uvicorn and python-dotenv. Note that audioop-lts is conditional on Python 3.13 or newer, so the interpreter version you choose changes the resolved set.

Editorial conclusion

Adopt AIAvatarKit if you are building a conversational avatar in Python and want to swap VAD, STT, LLM and TTS without rewriting transport code. Do not adopt it if you want a hosted product with no local services, or if you cannot run a VOICEVOX-compatible server, since the built-in application expects one at http://127.0.0.1:50021. Before committing, verify that your chosen STT and TTS providers are in the shipped implementation table, and read documents/getting-started.md for the CLI options and every setting of the built-in application.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. uezo/aiavatarkit on GitHub