# AVTR-1: a self-hosted flow-matching avatar renderer for live dialogue

> AVTR-1 renders lip-synced speech and active listening from a portrait image and dual-stream audio at 25 fps on one GPU. The weights and inference code are public, the production back end is not.

**avaturn-live/avtr-1** — AVTR-1: Avatars that listen back

- Repository: https://github.com/avaturn-live/avtr-1
- Stars: 497 · Forks: 78
- Language: Python
- License: NOASSERTION
- Published: 2026-09-20 · Updated: 2026-09-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/avaturn-live-avtr-1

## What AVTR-1 solves, and for whom

Most talking-head pipelines are one-directional. You hand them an audio track, they hand back a video of a face moving in sync with it. That is enough for a narrated clip and useless for a call, because a call has two audio streams and the interesting behaviour sits in the one the avatar is not speaking. AVTR-1 is built around that second stream. The README describes it as a flow-matching-based autoregressive model for live dialogue that takes a portrait image plus dual-stream audio and renders both lip-synced speech and active listening at 25 fps on a single GPU.

The intended audience is narrow and specific. The prerequisites list Linux, an NVIDIA GPU (Ampere or later recommended), CUDA 12.x and TensorRT 10.x. Nothing here targets a laptop CPU or a browser-only workflow. This is a project for engineers who already own inference hardware and want the renderer on it rather than in someone else's cloud. The README also advertises a managed API at avaturn.live and a hosted demo, which means the maintainers expect two classes of user: people who will never touch the code, and people who will run the whole stack themselves.

The README's own checklist is the honest summary of where the project stands. Model weights, inference code and an interactive streaming demo are marked done. A technical report and a production-ready back end are marked coming soon. If your requirement is a finished service, the repository does not currently contain one.

## Dual-stream audio and the two-pass dialogue render

The mechanism that distinguishes AVTR-1 shows up most clearly in the offline generation flags. A single-speaker run takes `--speech` and nothing else: the avatar lip-syncs the given track. A two-speaker run adds `--listen`, and the README states that the avatar voices `--speech` while reacting to the peer audio on `--listen`. The model is therefore conditioned on both what the avatar says and what it hears, which is what makes active listening a first-class output rather than a post-processed idle animation.

The consequence is that one render produces one side of a conversation. To get both sides you run the command twice with the tracks swapped, then combine the two files. The README gives the ffmpeg invocation for that, using `hstack` to place the videos side by side and `amix` to merge the audio. That is a deliberate design choice, not an oversight: each pass is a separate autoregressive render, and the project does not attempt to model both participants inside a single graph.

The weights come from two public HuggingFace repositories. AVTR-1 weights come from avaturn-live/avtr-1; the LivePortrait weights are repackaged as ONNX graphs from digital-avatar/ditto-talkinghead. That second dependency is worth noticing, because it means part of the rendering behaviour is inherited from a repackaged upstream model rather than authored here. The README does not explain how much of the final frame the LivePortrait component is responsible for.

## Installing AVTR-1 with pixi and building TensorRT engines

The project uses pixi as its environment manager rather than pip or conda directly. The README gives the install script for pixi itself, then a three-command setup. The clone and install steps are:

```bash
curl -fsSL https://pixi.sh/install.sh | sh
git clone https://github.com/avaturn-live/avtr-1.git
cd avtr-1
pixi install
```

After `pixi install` completes, the environment is resolved from pixi.toml. Note that pyproject.toml deliberately leaves the runtime dependency list empty, with a comment explaining that dependencies are managed by pixi.toml so an editable install does not fight the pixi-resolved environment. Do not expect `pip install -e .` to pull anything useful on its own.

Storage location is optional and controlled by one environment variable. The README states that downloaded weights and built engines go there, and that it defaults to `<project_root>/artifacts/` when unset:

```bash
export AVTR1_LOCAL_STORAGE=/path/to/avtr1_storage
```

The default is the repository checkout, not the caller's working directory, which matters if you run the commands from somewhere else and then wonder where the files went. Weights are fetched next. The README notes that the first run prompts for a HuggingFace login via `hf auth login`, invoked automatically as a dependency of the download task:

```bash
pixi run download
```

TensorRT engines are the step that ties the installation to the machine. The README states plainly that engines are compute-capability specific and built locally, and that the scripts should be run once per machine. You can build everything or build the three components individually:

```bash
pixi run build-trt-engines
pixi run build-trt-engines-avtr1
pixi run build-trt-engines-renderer
pixi run build-trt-engines-hubert
```

With engines in place, the interactive demo is a single task, and offline generation is where the flags described earlier live. This is the shortest path to a real output file:

```bash
pixi run interactive-demo
pixi run generate_offline --speech example/speaker_1.ogg
pixi run generate_offline --speech example/speaker_1.ogg --listen example/speaker_2.ogg --avatar elena --out elena.mp4
```

The repository ships example/speaker_1.ogg and example/speaker_2.ogg, so the first command works without any audio of your own. Passing `--avatar maria --bg minimal_office` selects a different reference frame and background. The README says available avatars are the filenames without `.png` inside `$AVTR1_LOCAL_STORAGE/v1/avatars_artifacts/reference_frames/` after downloading, which is how you discover what `--avatar` accepts. A run with no audio at all, `pixi run generate_offline --duration 10`, renders idle micro-motion for the given duration.

## The production back end is not in this repository

The clearest limitation is stated by the project itself: the production-ready back end is marked coming soon in the What's included list. What ships today is weights, inference code and a demo. The .env.example file describes a local-stream setup with a conversation engine and WebRTC ICE configuration, and it references README section 6 for a full walkthrough, but the README text available here is truncated before that section. Treat the streaming path as documented elsewhere and unverified from the main page.

The streaming configuration that is visible carries its own constraints. The OpenAI Realtime API key is required for the `openai` engine, and the .env.example notes that the streamer mints short-lived client secrets from it so the long-lived key is never sent to the browser. Cartesia is listed as an optional alternative, requiring CARTESIA_API_KEY and CARTESIA_AGENT_ID. TURN is optional too, and the file explains the failure mode precisely: ICE tries direct UDP first and only falls back to a relay when the network between browser and streamer cannot pass UDP, typical when the streamer runs on a cloud VM whose security group blocks inbound UDP. On a laptop or LAN setup those variables can stay unset.

Hardware is the other hard boundary. Ampere or later is recommended, CUDA 12.x and TensorRT 10.x are required, and engines must be rebuilt per machine. If your fleet is mixed, you are running the build scripts more than once. If you are on macOS or Windows, the prerequisites list Linux and there is no documented alternative.

## How AVTR-1 differs from Ditto-talkinghead and MuseTalk

Ditto-talkinghead is the closest reference point, and not only because AVTR-1 pulls LivePortrait weights repackaged from digital-avatar/ditto-talkinghead. Ditto targets audio-driven talking-head generation from a source video, which means the driving signal is a single audio track and the output is a rendered face. AVTR-1 adds the listening channel and the autoregressive, flow-matching framing, and it ships a live-session backend path with a conversation engine and WebRTC transport rather than only a generation script. The overlap in weights means the visual style of the renderer component is not entirely independent of that lineage.

MuseTalk is the other common comparison for anyone building lip-sync at low latency. MuseTalk is typically run as a diffusion-based lip-sync stage applied to an existing video, which keeps the rest of the frame untouched and makes it easy to bolt onto an existing pipeline. AVTR-1 instead renders the whole portrait from a reference frame and expects the dual-stream audio contract, so adopting it means adopting its avatar and background model, not just a mouth region.

Both alternatives avoid the TensorRT engine build, which is the operational cost unique to AVTR-1 in this comparison. That build buys the 25 fps single-GPU figure the README claims. If you do not need that throughput, the engine step is pure overhead.

## Licensing across four files and a patent notice

The repository does not carry a single licence. The top level contains LICENSE.md plus LICENSE-MODEL.md, LICENSE-RENDERER.md, LICENSE-STREAMER.md, THIRD-PARTY-NOTICES.md and PATENTS.md. GitHub reports the licence as NOASSERTION, which means the platform could not map the contents to a recognised identifier. The practical reading is that different parts of the stack are governed by different terms, and the model weights, the renderer and the streamer may not share the same permissions.

There is also a PATENTS.md file, which is unusual enough to be worth opening before any commercial deployment. The presence of THIRD-PARTY-NOTICES.md is consistent with the repackaged LivePortrait weights coming from another HuggingFace repository, since those carry their own upstream terms. This article cannot tell you what any of those files permit; read them, and take legal advice if the answer matters to your business. What can be said from the repository layout alone is that a single-line licence summary would be wrong here.

## Maintenance cadence and the cost of staying current

The last push to the default branch was on 2026-05-26. There are no retrieved releases, so there is no version history to track and no changelog to read. Upgrades therefore mean pulling the main branch and re-resolving the pixi environment, with no release boundary to pin against.

The upgrade cost is dominated by the TensorRT engines. Because they are compute-capability specific and built locally, a change to the model or the renderer can invalidate them, and the rebuild is per machine. A fleet of heterogeneous GPUs multiplies that work. The pixi.lock file is the one artefact that makes an upgrade reproducible, so pinning to a lockfile revision is the practical way to control when the rebuild happens rather than discovering it mid-deployment.

Storage growth is the quieter cost. Weights and engines both land under `$AVTR1_LOCAL_STORAGE`, and the README's default puts them inside the repository checkout, which means a clone can grow well past the size of the source tree. Pointing `AVTR1_LOCAL_STORAGE` at a dedicated volume before the first download avoids that.

## Conclusion

Adopt AVTR-1 if you have an Ampere-or-later NVIDIA GPU, a Linux host, and a reason to keep avatar rendering inside your own network; the offline generation path is the part that exists today. Do not adopt it if you need a shipped production back end, a Windows or macOS workstation, or a permissive licence you have not read: the repository splits rights across LICENSE-MODEL.md, LICENSE-RENDERER.md, LICENSE-STREAMER.md, LICENSE.md and PATENTS.md, and the GitHub licence field reports NOASSERTION. Before you commit, clone the repository, run pixi install and pixi run download, and confirm that the TensorRT engines build on your specific compute capability, because that build is the step most likely to fail on hardware the maintainers did not target.

## FAQ

### What does AVTR mean in AVTR-1?

The repository does not spell out the acronym. The README uses AVTR-1 as the name of the model and the weights repository, and the GitHub organisation is avaturn-live, so the name is tied to the Avaturn project rather than defined as a standalone term.

### Where do I get the AVTR-1 weights?

The README states that weights are pulled from two public HuggingFace repositories by the download step: AVTR-1 weights from avaturn-live/avtr-1, and LivePortrait weights repackaged as ONNX graphs from digital-avatar/ditto-talkinghead. The first run prompts for a HuggingFace login via hf auth login, invoked automatically as a dependency of the download task.

### What hardware does AVTR-1 require?

The prerequisites list Linux, an NVIDIA GPU with Ampere or later recommended, CUDA 12.x and TensorRT 10.x. TensorRT engines are compute-capability specific and built locally, so the build scripts must be run once per machine.

### Can AVTR-1 render both sides of a two-person conversation?

Not in a single pass. The README says to run generate_offline twice with the speech and listen tracks swapped, then combine the two outputs, and it gives an ffmpeg command using hstack and amix to produce one side-by-side video.

### Does AVTR-1 ship a production back end?

No. The README's What's included list marks the production-ready back end as coming soon, alongside the technical report. Model weights, inference code and an interactive streaming demo are the parts marked as available.

## Sources

- [avaturn-live/avtr-1 on GitHub](https://github.com/avaturn-live/avtr-1)
- [Issues](https://github.com/avaturn-live/avtr-1/issues)
- [README](https://github.com/avaturn-live/avtr-1/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/avaturn-live-avtr-1