LiveKit Agents: A Python framework for building server-side voice AI participants

What problem it solves and who it targets

LiveKit Agents addresses the gap between a plain LLM API call and a realtime voice conversation. If you want an agent that can hear a user, decide when to respond, speak back, and potentially see video, you need more than a chat loop. You need audio capture, speech recognition, turn detection, text generation, speech synthesis, and a way to transport all of that over a network with low latency. LiveKit Agents packages these pieces into a Python framework. The intended user is a developer building a voice AI product, not someone experimenting with a single LLM call. The README describes it as a framework for realtime, programmable participants that run on servers. That means the agent is not embedded in a mobile app or browser; it lives on a backend and communicates with clients through LiveKit's WebRTC infrastructure.

Architecture: Agent, AgentSession, and AgentServer

The core concepts are three distinct layers. An Agent is an LLM-based application with defined instructions. An AgentSession is the container that manages interactions between the agent and end users. The AgentServer is the main process that coordinates job scheduling and launches agents for user sessions. This separation matters. The AgentServer can handle multiple sessions, dispatching jobs to agents as users connect. The entrypoint function, decorated with @server.rtc_session(), acts like a request handler in a web server. It receives a JobContext and sets up an AgentSession for that specific interaction. This design lets you scale horizontally: you run an AgentServer, it accepts incoming RTC sessions, and each session gets its own AgentSession with its own VAD, STT, LLM, and TTS configuration. The README also mentions built-in task scheduling and distribution via dispatch APIs, which connect end users to agents without you writing custom queueing logic.

How the realtime loop works

The framework does not force a single model provider. The README states you can use any combination of STT, LLM, TTS, or a realtime API. In the simple voice agent example, the AgentSession is created with vad=inference.VAD() and then a combination of providers. Separately, the framework's semantic turn detection uses a transformer model. This is not a simple energy threshold. The documentation describes it as detecting when a user is done with their turn, which helps reduce interruptions. That is a meaningful difference from naive silence-based detection. The agent can also use function tools. The example shows a @function_tool decorator on a lookup_weather function, which the LLM can call during the conversation. The data flow is: audio comes in from the WebRTC session, VAD decides when the user stops talking, STT converts speech to text, the LLM processes the text and may invoke tools, TTS converts the response to speech, and the audio goes back over the WebRTC connection. The framework handles the orchestration of this loop, including the timing of when to generate a reply.

Getting it running: installation and environment

Installation is a standard pip command. The README shows: pip install "livekit-agents[openai,deepgram,cartesia]". That command pulls the core library plus plugins for OpenAI, Deepgram, and Cartesia, covering LLM, STT, and TTS respectively. You are not limited to those. The repository's pyproject.toml lists many more plugin packages, such as anthropic, assemblyai, aws, azure and elevenlabs. After installation, you need three environment variables: LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. These point to a LiveKit server, which you must run yourself or provision. The README points to the open-source LiveKit server as one of the most widely used WebRTC media servers, and links to its separate repository. The example code is minimal: define a function tool, create an AgentServer, define an entrypoint, and run it via the cli module. The entrypoint creates an AgentSession with a VAD and provider combination. The framework also supports multi-agent handoffs, as shown in an abbreviated example where an IntroAgent gathers information and then presumably hands off to another agent. That example uses an on_enter method and a generate_reply call with custom instructions.

Built-in testing framework for non-deterministic agents

Testing voice agents is hard because LLM outputs are not deterministic. LiveKit Agents includes a native test integration. The README shows a pytest example where you create an AgentSession with a specific LLM, start it with your agent, and then run a user input. The result object has an expect chain. You can skip events, check for function calls, and verify function call outputs. The example checks that after a user says they need to place an order, the next event is a function call named start_order, then a function call output, then an assistant message. This is a concrete way to assert agent behavior without relying on exact text. The test framework also supports judges, which the README says help ensure your agent is performing as expected. This testing capability is a genuine differentiator.

Limitations and cases where it is the wrong tool

The framework is tied to LiveKit's ecosystem. If you are not already using LiveKit for WebRTC, you must adopt it, including running a LiveKit server. That is a significant operational dependency. The server lives in its own repository, github.com/livekit/livekit. Another limitation is the reliance on plugins. The README shows a specific set of providers, and while the framework is flexible, you will need to verify that the plugin for your preferred STT or TTS exists and is maintained. The semantic turn detection is a transformer model, which adds compute overhead and may not be suitable for very low-resource deployments. The framework is also Python-only; there is a separate AgentsJS library for JavaScript, which means you cannot share agent logic across languages. For simple text-based chatbots that do not need realtime audio, this framework is overkill. You would be better off with a direct LLM API call.

Alternatives and how they differ

The most direct alternative is building on a realtime API directly, such as OpenAI's Realtime API, which provides speech-to-speech without separate STT, LLM, and TTS components. LiveKit Agents lets you mix and match providers, but a realtime API bundles them into a single model. That is a trade-off: bundling simplifies the pipeline but reduces flexibility. Another alternative is a framework like Pipecat, which also targets voice agents but is not tied to LiveKit's infrastructure. The key difference is that LiveKit Agents integrates deeply with LiveKit's job dispatch and telephony stack. If you need SIP calling, LiveKit's telephony integration is a reason to choose this framework. If you want to avoid LiveKit entirely, you would need to handle WebRTC yourself or use a different media server. The README also mentions MCP support, allowing you to add tools from MCP servers with one line of code, which is a modern integration point that some alternatives lack.

Maintenance, licensing, and upgrade cost

The repository is under the Apache-2.0 license, which permits commercial use, modification, and distribution without a copyleft requirement. The last push was November 2, 2026, and recent releases are 1.8.10, 1.8.9 and 1.8.8, all in October 2026. That indicates active maintenance. The upgrade cost will depend on how much you rely on the framework's internal APIs. The example code uses high-level abstractions like AgentSession and function_tool, which are likely stable, but the testing API and plugin interfaces may change between minor versions. The README recommends installing the LiveKit Docs MCP server and the Agent Skill for AI coding assistants, which suggests the project expects developers to use AI tools to navigate its API. That is a sign that the API surface is large and not fully documented in the README alone. You should budget time to read the full documentation at docs.livekit.io/agents before adopting it for a production system.

Editorial conclusion

Adopt LiveKit Agents if you are building a realtime voice agent that must integrate with LiveKit's WebRTC infrastructure, need built-in job scheduling and telephony support, and want to mix STT, LLM, and TTS providers. Skip it if you only need a simple chatbot without realtime audio, or if you cannot run a LiveKit server and its dependencies. Before committing, verify that your target model providers have plugins, that your deployment can handle the WebRTC and job dispatch overhead, and that the built-in test framework covers your expected interaction patterns. The framework is Apache-2.0 licensed, so you can run the entire stack on your own servers, but you must accept the operational cost of running LiveKit itself.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes