TEN Framework: A Graph Runtime for Real-Time Voice Agents

The gap TEN fills between a demo script and a voice product

A voice agent is not one model. It is a microphone, a speech-to-text service, a language model, a text-to-speech service, and something that decides when the user has stopped talking. Wiring those together is where most projects stall, because the pieces come from different vendors with different streaming semantics. TEN's answer is to make the whole chain a graph of components it calls extensions, and to ship a set of working graphs rather than a blank page. The README lists the ecosystem as the framework itself plus Agent Examples, VAD, Turn Detection, and Portal, which tells you the intended audience: developers who want to assemble and then replace parts of a real-time conversation loop. The examples named in the README are concrete about the range: a multi-purpose voice assistant with RTC and WebSocket transports, a Doodler that turns speech into sketches, speaker diarization, lip-sync avatars, SIP calling, transcription, and an ESP32-S3 Korvo V3 hardware target. That last one is the clearest statement of intent. This is not a library you call from a web handler. It is a runtime that can sit on a development board.

Extensions, graphs, and where the audio actually goes

The repository describes TEN as an open-source framework for real-time multimodal conversational AI, and the examples are organised as agents that you build and then serve. The README's own vocabulary is the useful part: an agent is built from extensions, and the voice assistant example can be extended with Memory, VAD, and Turn Detection. Those three additions map onto the hard problems in voice. VAD decides whether there is speech in the frame. Turn Detection decides whether the user has finished a thought, which is a different question and the one that causes agents to interrupt people. Memory is what survives the turn. The README also states that the assistant supports both RTC and WebSocket connections, so the transport is a choice rather than a fixed assumption. What the supplied material does not give is the internal message format between extensions, the scheduling model, or how backpressure is handled when the language model is slower than the audio stream. Those are the details that determine whether a graph runtime is pleasant or painful, and they are not in the README. Treat the architecture as something to read in the documentation and source before you commit, not something this review can confirm.

Getting the voice assistant running in Docker

The quick start is explicit and assumes Docker. You clone the repository, change into ai_agents, and copy the environment template: cd ai_agents, then cp ./.env.example ./.env. The .env file expects five keys: AGORA_APP_ID, AGORA_APP_CERTIFICATE, DEEPGRAM_API_KEY, OPENAI_API_KEY, and ELEVENLABS_TTS_KEY. The README annotates them: Deepgram is required for speech-to-text, OpenAI for the language model, ElevenLabs for text-to-speech. Agora supplies the real-time transport, which is why its App ID and App Certificate sit alongside the model keys rather than in a separate section. From there, docker compose up -d starts the development containers, docker exec -it ten_agent_dev bash puts you inside, and you pick an example: cd agents/examples/voice-assistant for the chained assistant, or cd agents/examples/voice-assistant-realtime for the speech-to-speech variant. The README estimates roughly five to eight minutes for the default build, and notes that you run task install before the first launch. The stated minimums are modest: two CPU cores and four gigabytes of RAM, plus Node.js LTS v18 on the host.

Four vendor keys before you hear anything

The prerequisite table is the most honest part of the README, and it is also the main limitation. Before the agent says a word you need accounts with Agora, OpenAI, Deepgram, and ElevenLabs. Each of those is a separate signup, a separate quota, and a separate failure mode. If Deepgram returns errors, the agent is deaf. If ElevenLabs is slow, the agent is mute regardless of how fast the language model is. The README presents these as required, not as defaults you can swap, so the quick-start path is effectively a fixed vendor stack even though the framework's design implies the pieces are replaceable. That distinction matters. A framework that can swap components is not the same as a template that ships with four hard dependencies, and the README does not walk through replacing any of them. The second constraint is Agora. Its App ID and App Certificate are listed under Keys for the localhost quick start, which means the default path routes through Agora's real-time network. If your deployment cannot use that transport, you are off the documented path on day one. The README mentions a WebSocket example, so an alternative exists, but the quick start does not use it.

Where a graph runtime is the wrong tool

TEN is a poor choice when the conversation is not the product. If you need to transcribe a batch of recorded calls, the transcription example exists, but you are paying for a real-time runtime and its container orchestration to do an offline job that a single API call handles. If your agent is text-only, the voice and video machinery is overhead. If you need a hosted endpoint with an SLA, TEN gives you a repository and a Docker Compose file, not a service. The ESP32 example is the clearest counterexample in the other direction: running the agent on an ESP32-S3 Korvo V3 board is something a hosted API cannot do at all, and it is the case where owning the runtime pays for itself. There is also a maintenance question the README does not answer. The release list shows three releases in the 0.11.x line within roughly five weeks in mid-2026, which suggests active iteration. Frequent point releases are good for fixes and awkward for anyone pinning a version, and the material gives no compatibility policy or deprecation window. For a framework whose value is a graph of third-party extensions, that policy is the thing you would want to read next.

TEN against Pipecat and LiveKit Agents

The closest comparisons are Pipecat and LiveKit Agents, both of which also build real-time voice pipelines from swappable components. The difference is where the centre of gravity sits. LiveKit Agents is built around LiveKit's own WebRTC infrastructure, so if you are already running LiveKit for video, the agent is a participant in a room you control. TEN's default path uses Agora for transport, which puts a different real-time vendor at the centre and makes the agent a participant in Agora's network instead. Pipecat is closer to a Python library for composing frames, with less emphasis on a packaged agent runtime and a hardware target. TEN's distinguishing move is the ecosystem around the core: the README names VAD, Turn Detection, and Portal as separate projects, and the examples span avatars, SIP, and an ESP32 board. That breadth is the argument for TEN over a thinner pipeline library. It is also the argument against it, because each of those surfaces is another thing to keep current. If your requirement is a WebRTC agent inside an existing LiveKit deployment, TEN's Agora-centric quick start is friction you do not need. If your requirement is a voice agent that can also run on a microcontroller or answer a SIP call, the alternatives are thinner.

Licence, versioning, and what to check before forking

The repository metadata reports the licence as NOASSERTION, which means GitHub could not match the licence file to a known identifier. That is not the same as having no licence, and it is not the same as a permissive one. Until you read the actual file, you cannot assume the terms under which you may redistribute a modified agent, and the examples include avatar vendors and telephony integrations that may carry their own terms. This is a reading task, not a legal opinion, and it should happen before you build a product on the code rather than after. The same caution applies to the release cadence. Three releases in the 0.11.x series between late July and early August 2026, with the repository last pushed in September 2026, indicates a project that is still moving. Pin a version in your own build, and check the release notes for interface changes to the extensions you depend on. The README's own instruction to run task install before the first launch and after updates is a small signal that the build step is not idempotent by default and that upgrading is an operation you perform deliberately.

A concrete first hour with TEN

The fastest way to decide is to run the default example and watch where it breaks. Clone the repository, cd ai_agents, copy .env.example to .env, fill in the five keys, run docker compose up -d, then docker exec -it ten_agent_dev bash. Inside the container, cd agents/examples/voice-assistant, run task install, and start the web server as the README describes. The five-to-eight-minute build estimate is the first thing to measure, because it tells you how expensive your edit-test loop will be. The second thing to measure is latency with Turn Detection enabled versus disabled, since that extension is the one most likely to change how the agent feels. If the default stack works and you can see where to swap a component, TEN is worth the four vendor accounts. If you find yourself fighting the container to change the speech-to-text provider, the framework's flexibility is not reaching the surface you need, and a thinner pipeline library will cost you less. That test takes an afternoon and answers the question the README cannot.

Editorial conclusion

Adopt TEN if you are building a voice product where the pipeline itself is the product: custom turn detection, a specific ASR vendor, an avatar, or a phone line. Do not adopt it if you want a managed endpoint and have no interest in operating Docker and API keys. Before committing, verify three things: the terms of the NOASSERTION licence file, whether the Agora App ID requirement fits your transport plans, and whether the example you intend to fork still matches the interface in the current release notes.

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. TEN-framework/ten-framework on GitHub