# Gander: a duplex voice agent that keeps working while you talk

> Gander is an open source model and runtime for natural full-duplex voice collaboration, built so a conversation can continue while a long task runs in the background. Its architecture splits a streaming Cerebellum for real-time interaction from a pluggable Back Brain for reasoning, and the project ships a paper, a model on Hugging Face and a dataset that is still being released.

**Omni-Interaction-Gander/Omni-Interaction-Agent** — Open-source end-to-end omni interaction agent for natural full-duplex voice collaboration, continuous audio-visual perception, and asynchronous long-horizon agentic execution.

- Repository: https://github.com/Omni-Interaction-Gander/Omni-Interaction-Agent
- Website: https://omni-interaction-gander.github.io/Omni-Interaction-Agent/
- Stars: 556 · Forks: 34
- Language: Python
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/omni-interaction-gander-omni-interaction-agent

## The problem the design solves: talking and working at the same time

Gander is described as an open source end to end omni interaction agent for natural full-duplex voice collaboration, continuous audio-visual perception, and asynchronous long-horizon agentic execution.

The tagline in the project page captures the intent in one line: talk while it works. You can interrupt it, redirect it, or show it what you see without stopping the task.

That last phrase is the differentiator. Most voice assistants are turn based. You ask, it works, it answers. The alternative is an assistant you delegate a task to and then keep talking while it runs, refining requirements, asking follow-up questions, checking progress, granting permissions or cancelling, all through the same conversation.

The overview frames Gander as a native duplex interaction model with an asynchronous agent loop, and describes it as trained for spoken collaboration, combining natural expressive dialogue with knowledge and instruction-following capabilities.

The system design is the Cerebellum-Brain architecture, pairing a streaming conversational model with an asynchronous reasoning agent so that responsive interaction and long-running execution live in the same conversation.

The project is Apache-2.0 licensed and written in Python, with a paper on arXiv, a model on Hugging Face, a project page, a demo video and a dataset that is still being released. The last push was on 2026-09-17 and there are no tagged releases, and the news section records the project being officially released on 2026-09-09.

## Three modules and one causal path through a request

The architecture table is the clearest statement of the design, with three modules and a clear division of responsibility.

The Streaming Cerebellum, implemented in minicpm_ft, owns realtime audio-visual perception, interaction control, text generation and speech synthesis. The Agent Orchestration Runtime, in gander_runtime, handles trusted-turn binding, task state, multimodal context, worker events, permissions and delivery. And the Pluggable Back Brain sits behind a provider interface and takes on long-horizon reasoning, tool use, file and application operations, and workflow execution.

The framing sentence in the project page is that the Cerebellum owns realtime interaction, the Brain owns long-horizon work, and the runtime connects both on one task timeline.

A live request follows four steps, and each step is a design decision rather than a description of plumbing.

The browser streams microphone audio and optional camera or screen frames, and source timestamps align visual evidence with the audio captured at that time, so the model knows what it saw when it heard what you said. The Cerebellum then predicts whether to listen, speak, interrupt, or invoke a task operation, and simple requests are answered locally without ever touching the Brain. When a task does need doing, the runtime binds the agentic work to the finalized user turn and starts a tracked Brain execution with bounded text and visual context. The Brain then works asynchronously, and its milestones, questions, permissions and final results come back through the same live conversation.

That fourth point is what makes the whole thing coherent. The Brain does not report back through a separate channel, so the user never has to watch a spinner and then read a result. The conversation is the interface for a running job.

One detail is worth noting for anyone comparing this to a speech-to-speech stack. Managed ASR supplies the browser transcript and the trusted text instruction passed to the Brain, while the Cerebellum itself consumes raw audio directly. So the conversation model is not working from a transcript, while the reasoning model is.

## The Cerebellum: one second units and a control token before content

The Streaming Cerebellum starts from MiniCPM-o 4.5 and flattens continuous interaction into one-second causal units, and the pipeline is shown in the README as:

```text
[video] [audio] [optional task context] -> [control] [text or tool content]
```

The control token is predicted before content, and the README explains why: that separates whether to act from what to produce.

That ordering is what makes interruption cheap. A model that generates the response text first has to decide what to say before it can decide whether to say it, so bailing out mid-utterance means throwing away work. Predicting a control decision first means the decision to speak and the decision to stay silent are both cheap, and the expensive part, the text, only runs when the control token says to.

The control vocabulary has four values. listen means stay silent and continue observing the stream. speak means generate a bounded text segment and send it to the Talker. interrupt means stop an utterance that is no longer appropriate, including when the user takes the floor. And tool means emit a structured task operation for the runtime.

The speech path has a second separation. The Cerebellum itself has a Thinker-Talker structure, where the Thinker controls interaction and content while the detached Talker renders speech without blocking perception. On a speak unit, the Talker conditions on Thinker hidden states and aligned text tokens to predict S3 speech tokens, and a causal flow-matching decoder renders audio incrementally. The released alignment is eight text tokens to 50 S3 tokens per unit.

The word detached is doing real work in that sentence. Detached deployment lets perception continue while speech is generated, which is the requirement for a conversation where the user can talk over the assistant and be heard while it is still talking.

For long sessions the model retains up to 128 recent units, and the runtime supports three context modes: context_no_previous, context_slate and context_memory. The release serving profile uses context_slate, which preserves active task state without requiring a memory model.

## The runtime, the task vocabulary and the state that outlives a turn

The Agent Orchestration Runtime is where the Cerebellum's small vocabulary becomes a full execution lifecycle.

The Cerebellum exposes a small task vocabulary, and the runtime turns it into the operations that a task can go through. The README begins to enumerate them, with the first listed as task_st..., and the table is cut off in the text available here, so the complete operation set is not visible without opening the project page.

What the surrounding description does establish is the shape of the problem. Users can keep talking or revise a task while execution generations prevent stale work from becoming the final answer. That phrase, execution generations, is the interesting one, because it addresses a failure mode that any async agent system has. If a user says wait, actually use the blue version instead, halfway through a long task, the work already in flight was started under the old instruction. Without generation tracking, the stale result can arrive after the corrected one and overwrite it, or be presented as the answer to a question the user has already moved on from.

The runtime's responsibilities, as the table lists them, are trusted-turn binding, task state, multimodal context, worker events, permissions and delivery. Trusted-turn binding is the mechanism that decides which words of a conversation become the instruction for the Brain, as opposed to chatter. The Cerebellum makes that call when it emits a tool operation, and the runtime binds the agentic work to the finalized user turn, so the instruction the Brain receives is the one the model decided was a real request.

Permissions sit in the same place, which makes sense given the design. A long-running agent may reach a point where it needs to grant access to a file or an application, and the README says those permission requests come back through the live conversation rather than through a separate approval dialog. So the user is asked while they are already talking, in the same thread, and the task waits.

The overall picture is a system where the conversation is the only control surface. The user is not presented with a separate agent interface, and the runtime is what makes that possible.

## What is available now, and what is not

The resources section of the README lists what a reader can actually get hold of, and there is one item marked as pending.

There is the paper, on arXiv with the identifier 2609.08977. There is the project page, at the GitHub Pages site Omni-Interaction-Gander.github.io, which is also the repository homepage. There is the model on Hugging Face at Gander-Omni/Gander. And there is the dataset, whose badge says coming soon.

The news section explains why. On 2026-09-09 Gander was officially released, and on 2026-09-10 the dataset was still undergoing the open source release process and would be made publicly available soon.

So at the point the README was written, the system was open source in the sense that the code, the paper and the weights were available, but the training data was still in the release process. That matters for anyone planning to train or fine-tune rather than to run the released model, because the data is the part you need for the former and can do without for the latter.

The demo is a video on YouTube, and the capabilities image in the docs assets directory is described as covering live audio-visual interaction and agentic tasks.

The repository itself is small and organised around the two components the architecture table names. There is a gander_runtime/ directory for the orchestration runtime, a minicpm_ft/ directory for the streaming Cerebellum implementation, a scripts/ directory, a docs/ directory, and an environment.yml at the root, which is the conda environment file for the dependencies. There is no setup.py or pyproject.toml in the listing, so the project is set up through the conda environment rather than through a Python packaging toolchain, which is a reasonable choice for a research release with a specific set of model dependencies but does mean the install path is not the standard pip one.

## Conclusion

Adopt Gander if you are building a voice interface where a user has to keep talking while long work runs, since that is the specific problem the Cerebellum and Brain split is designed around. Do not adopt it expecting a turn-based assistant, because the whole design assumes interruption, overlap and redirection are normal. Before you plan a deployment, check the dataset release status, since the training data was still going through open source release in September 2026, and read the quick start and offline inference sections for the model hosting requirements.

## FAQ

### What is Gander and what problem does it address?

It is an open source end to end interaction model and runtime for natural full-duplex voice collaboration, continuous audio-visual perception and asynchronous long-horizon task execution. The point is that you can talk, interrupt, redirect or share your screen while a delegated task keeps running, with results returning through the same conversation.

### What is the Cerebellum-Brain architecture in Gander?

The Streaming Cerebellum, implemented in minicpm_ft, handles realtime audio-visual perception, interaction control, text generation and speech synthesis, while a pluggable Back Brain behind a provider interface handles long-horizon reasoning, tool use and workflow execution. The gander_runtime module connects the two on one task timeline.

### How does Gander handle interruption while it is speaking?

The Cerebellum predicts a control token before generating content, separating whether to act from what to produce, with control values for listen, speak, interrupt and tool. The Talker is detached from the Thinker, so perception continues while speech is generated, and a detached deployment lets the model hear the user take the floor.

### Is the Gander dataset available yet?

Not at the time the README was written. The news section dated 2026-09-10 says the dataset was undergoing the open source release process and would be made publicly available soon, while the code, the paper and the model on Hugging Face were already released on 2026-09-09.

### How does Gander handle a user changing their mind midway through a task?

The runtime uses execution generations so that stale work cannot become the final answer, and it binds agentic work to the finalized user turn so the instruction the Brain receives is the one the model judged to be a real request. Questions, permissions and results all return through the live conversation.

## Sources

- [Issues](https://github.com/Omni-Interaction-Gander/Omni-Interaction-Agent/issues)
- [License: Apache-2.0](https://github.com/Omni-Interaction-Gander/Omni-Interaction-Agent/blob/main/LICENSE)
- [Omni-Interaction-Gander/Omni-Interaction-Agent on GitHub](https://github.com/Omni-Interaction-Gander/Omni-Interaction-Agent)
- [Project website](https://omni-interaction-gander.github.io/Omni-Interaction-Agent/)
- [README](https://github.com/Omni-Interaction-Gander/Omni-Interaction-Agent/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/omni-interaction-gander-omni-interaction-agent