# StableAvatar: infinite-length audio-driven avatar video from a reference image

> The first end-to-end video diffusion transformer for talking avatars, with the audio pipeline, the sliding window fusion and the training scripts all published in one repository.

**Francis-Rings/StableAvatar** — [NeurIPS2026] We present StableAvatar, the first end-to-end video diffusion transformer, which synthesizes infinite-length high-quality audio-driven avatar videos without any post-processing, conditioned on a reference image and audio.

- Repository: https://github.com/Francis-Rings/StableAvatar
- Stars: 1,263 · Forks: 115
- Language: Python
- License: MIT
- Published: 2026-10-06 · Updated: 2026-10-06 · Language: en
- Canonical page: https://hysenlabs.com/projects/francis-rings-stableavatar

## What the paper claims, and where the claim is narrow

StableAvatar describes itself as the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing, conditioned on a reference image and audio. Two of those words carry the weight. End-to-end means no separate face restoration or face swapping pass afterwards, and the README is specific about which tools that rules out, naming face-swapping tools like FaceFusion and face restoration models like GFP-GAN and CodeFormer. Infinite-length means the claim is about duration rather than fidelity: the paper's complaint about prior work is not that its frames look bad, it is that they drift.

That framing matters for judging the project. Most avatar video work in 2025 was a pipeline: lip sync here, enhancement there, identity fix at the end. The pitch here is that the drift came from the audio side, and that fixing it inside the model removes the need for the cleanup stage entirely. The repository is Fudan University and Microsoft Research Asia work with collaborators at Xi'an Jiaotong University and Tencent, published as arXiv 2508.08248 with a project page, a Hugging Face model, a Space demo, and video comparisons on YouTube and Bilibili.

The news section records the pace: a public Space demo on 2025-08-29, ComfyUI support the week before, finetuning and LoRA code released on 2025-08-16, a fix for multi-GPU versus single-GPU inconsistency on 2025-10-12, and NeurIPS2026 acceptance noted on 2026-09-25. The repository is not archived, last pushed 2026-09-25, with 1262 stars and only 2 open issues.

## Why the audio path is the actual contribution

The interesting paragraph in the overview is the diagnosis, not the remedy. Existing audio-driven avatar models, the paper argues, rely on third-party off-the-shelf audio extractors, inject the resulting embeddings straight into the diffusion model through cross-attention, and suffer for it: because the diffusion backbone carries no audio prior of its own, the error accumulates across clips, so the latent distribution of later segments drifts away from where it should be.

Three components answer that. The Time-step-aware Audio Adapter modulates conditioning by diffusion time step, which is what stops error from piling up. The Audio Native Guidance Mechanism uses the diffusion's own evolving joint audio-latent prediction as a guidance signal at inference time, tightening audio synchronization without a separate synchronisation network. The Dynamic Weighted Sliding-window Strategy fuses latents across time at the seam, which is what makes the length indefinite rather than merely long.

Each of those three names maps to a distinct failure mode: temporal drift in identity, audio going out of sync, and a visible seam between windows. That the fixes are separated this cleanly, and each named after the problem it addresses, is a good sign about how the project was built. It also means a reader can ablate one component and measure its effect, which is not how most published avatar systems arrive.

## What the repository tree exposes about scale

The tree is the most honest documentation in the repository, because the filenames encode model sizes and training strategies side by side. There is `train_14B.py` and `train_14B.sh`, plus `train_14B_lora.py` and its shell wrapper. Then a 1B family that is more interesting than a plain size reduction: `train_1B_square.py`, `train_1B_rec_vec.py`, and LoRA variants of each, with separate 64-shard shell scripts for the recurrent-vector and LoRA runs.

Reading that as an experiment log rather than a menu: the authors appear to have used a 1B model to compare square frame layout against a recurrent vector layout, with and without LoRA, then scaled the winner to 14B. That is a far more informative arrangement than a single training script, and it costs a repository nothing to publish.

Inference follows the same pattern of separate entry points: `inference.py` with `inference.sh`, and `multiple_gpu_inference.sh` alongside it, which matches the news entry about multi-GPU and single-GPU results disagreeing until a patch was merged in October 2025. If you are reproducing anything, that divergence is the first thing to check, since it silently produces different video depending on how many GPUs you have.

The supporting modules are named plainly: `audio_extractor.py` for the audio front end, `lip_mask_extractor.py` for mouth region masking, `vocal_seperator.py` for isolating a voice from a mixed track, and `app.py` for the Gradio interface. The `wan/` directory, plus `accelerate_config/` and `deepspeed_config/`, say where the compute story comes from.

## The dependency floor is pinned hard

There is one requirements file and it is unusually prescriptive, which is good news for anyone planning to reproduce this. The pins that matter are the deep learning stack itself.

```python
torch==2.7.0
torchvision==0.22.0
diffusers==0.30.1
transformers==4.51.3
```

The numpy constraint is worth reading twice, because it is the kind of line that saves an afternoon.

```python
numpy>=1.23.5,<2
```

Beyond those, the file pulls in `timm`, `einops`, `safetensors`, `tomesd`, `accelerate`, `omegaconf`, `decord`, `librosa`, `torchaudio==2.7.0`, `moviepy==1.0.3`, `imageio-ffmpeg`, `opencv-python`, `albumentations`, `scikit-image`, `SentencePiece`, `tensorboard` and `gradio>=5.0.0`. `dashscope` is there too, the Alibaba Cloud model service client, presumably for a hosted audio or generation path. `ftfy` is listed twice, which suggests it was added twice rather than thought about twice.

The practical reading is that this is a research repository, not a packaged one. There is no setup script, no versioned release and no container. `torch==2.7.0` paired with `numpy<2` is a specific, dated combination, and on a machine where numpy 2 is already installed you will want a separate environment rather than a workaround.

## Community front ends arrived before the paper did

One detail says more about adoption than the star count does. Within days of the code landing in August 2025, third parties had built a ComfyUI node and a Gradio interface, and the ComfyUI contributor is credited by name in the news section. The ComfyUI path runs in 10 steps and is described as 3x faster, which is the single most practically useful number in the README.

That kind of front-end work changes what a research repository is good for. ComfyUI nodes exist so a model can be sampled and tuned without writing Python, which turns a paper into something you can try on your own reference image in an afternoon. A Gradio interface does the same for people who would rather upload an image and a wav file than clone a repository with pinned torch versions.

The hosting story is less comfortable. The public demo on Hugging Face Spaces was limited to Hugging Face Pro users because of the long video generation time, which is an honest admission that throughput is the limiting factor rather than a solved problem. The README's own sample output is a grid of looping videos embedded from GitHub user attachments, so the results you will see are curated rather than reproducible on demand.

## What the README does not tell you

There are no tagged releases, so there is no changelog to read and no version to pin in an install command. The repository topics are just three, `aigc`, `avatar-generator` and `video-generation`, which says the authors consider this a general generation project rather than a talking-head project specifically.

Inference cost is the missing number. The README says the demo is slow enough to be paid-only, says the ComfyUI path is faster in steps, and gives no hardware requirement, no expected generation time and no VRAM figure. For a 14B video diffusion model, anyone planning to run this locally has to make that guess themselves, and a 1B variant appears in the training scripts without a documented inference path of its own.

Training data is also absent from what the README covers. For a project whose central claim is identity preservation, the identity data it was trained on, and any consent or licensing position on it, is exactly what a reader would want and cannot find here. The arXiv paper and the project page are where that has to be answered, and the README serves as the pointer rather than the source. Six example directories, `examples/case-1/` through `examples/case-6/`, sit in the tree for inputs and outputs.

## Conclusion

StableAvatar is unusual among research video repositories because the interesting decisions were published rather than left inside a demo: the audio adapter, the joint audio-latent guidance, the dynamic weighted sliding window, and the 14B training scripts are all in the tree, so the design can be argued with instead of merely watched. What the README does not settle is anything about inference cost. Generation is slow enough that the hosted demo was restricted to paid users, there are no published releases to track, and the 10 step ComfyUI path is the only documented route to a faster turnaround. The dependency floor is also firm, with torch and torchvision pinned and numpy held below 2. Start from `inference.py` with a short audio clip, read the architecture section before touching the training scripts, and judge the identity consistency claim on your own reference image rather than on the sample videos.

## FAQ

### Does StableAvatar need a separate face restoration or face swapping step?

No, and that is the central claim. The README says the videos are directly synthesized without face-related post-processing tools, and names face-swapping tools like FaceFusion and face restoration models like GFP-GAN and CodeFormer as the kind of step the model is meant to remove. The paper frames this as end-to-end generation rather than a pipeline.

### How does StableAvatar generate video longer than the model was trained on?

A Dynamic Weighted Sliding-window Strategy fuses latents over time at the window boundary, which is what makes the length indefinite rather than fixed. Two other components address the drift that usually appears in long clips: a Time-step-aware Audio Adapter that modulates by diffusion time step, and an Audio Native Guidance Mechanism that uses the diffusion's own joint audio-latent prediction as a guidance signal at inference time.

### What does StableAvatar need to run inference, and how fast is it?

Expect a dated research environment rather than a packaged install. The requirements file pins torch 2.7.0, torchvision 0.22.0, diffusers 0.30.1 and transformers 4.51.3, and holds numpy below 2. On speed, the README notes the Hugging Face demo was restricted to Pro users because of long generation times, and points to a community ComfyUI node that runs in 10 steps, described as 3x faster.

### Can StableAvatar be finetuned on a specific person?

The repository publishes finetuning and LoRA training code, including `train_14B_lora.py` and a 1B LoRA script with a separate 64-shard runner, alongside full training scripts for both sizes. The news section records the finetuning release on 2025-08-16, but the README documents no training data, identity data provenance or hardware guidance, so the paper is where to look for those.

## Sources

- [Francis-Rings/StableAvatar on GitHub](https://github.com/Francis-Rings/StableAvatar)
- [Issues](https://github.com/Francis-Rings/StableAvatar/issues)
- [License: MIT](https://github.com/Francis-Rings/StableAvatar/blob/main/LICENSE)
- [README](https://github.com/Francis-Rings/StableAvatar/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/francis-rings-stableavatar