SoulX-LiveAct, real-time human animation video from audio and a still image

The three contributions, and what each one buys you

SoulX-LiveAct presents a novel framework that enables lifelike, multimodal-controlled, high-fidelity human animation video generation for real-time streaming interactions. The paper's summary is structured as three numbered claims, and reading them in order tells you what is actually new.

The first is that the authors identify diffusion-step-aligned neighbour latents as a key inductive bias for AR diffusion, providing a principled and theoretically grounded neighbour forcing for step-consistent AR video generation. The problem being solved is that autoregressive video generation produces frames that drift, because each step is conditioned on the previous one and small errors compound. Aligning the neighbours to the diffusion step is the mechanism that keeps a long sequence coherent.

The second is ConvKV Memory, a lightweight plug-in compression mechanism that enables constant-memory hour-scale video generation with negligible overhead. The word constant is the important one, because the alternative to compressing the attention key and value cache is that memory grows with sequence length until the run dies. A plug-in that keeps memory flat is what makes hour-scale generation possible at all, and the constant-memory property is what allows the two-GPU configuration to keep running.

The third is an optimised real-time system that achieves 20 FPS using only two H100 or H200 GPUs with end-to-end adaptive FP8 precision, sequence parallelism and operator fusion at 720 by 416 or 512 by 512 resolution. The performance comes from three things working together: reduced precision adaptively applied, the work split across two devices, and kernels fused so the memory traffic is not the bottleneck.

The authors listed are Dingcheng Zhen, Xu Zheng, Ruixin Zhang and Zhiqi Jiang, with Yichao Yan, Ming Tao and Shunshun Yin also named. The technical report is on arXiv, and there is a project page, a GitHub home and a Hugging Face model card.

Installing it, and the three compiled dependencies that matter

The installation is four steps and three of them compile something from source.

Step one is the environment, a conda environment on Python 3.10 with the requirements installed and sox from conda-forge:

conda create -n liveact python=3.10
conda activate liveact
pip install -r requirements.txt
conda install conda-forge::sox -y

Step two is SageAttention, which is needed to enable the FP8 attention kernel. The pinned version is v2.2.0:

git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention
git checkout v2.2.0
python setup.py install

And there is an optional fork, a modified version of SageAttention, which is needed to enable operator fusion for the QKV operations:

git clone https://github.com/ZhiqiJiang/SageAttentionFusion.git
cd SageAttentionFusion
python setup.py install

Step three is vllm, needed to enable the FP8 GEMM kernel, pinned exactly:

pip install vllm==0.11.0

Step four is LightVAE:

git clone https://github.com/ModelTC/LightX2V
cd LightX2V
python setup_vae.py install

So the accuracy claims are not free. The FP8 attention needs SageAttention, the operator fusion needs a specific fork of it, the FP8 GEMM needs a pinned vllm, and the VAE needs a separate install. The requirements.txt shows the rest of the stack, and it is a research stack rather than a library one: torch 2.8.0 with matching torchaudio and torchvision, diffusers and transformers for the model layers, accelerate, xfuser 0.4.5 for the parallel diffusion, gradio for the GUI demo, and then the media handling, imageio-ffmpeg, moviepy 1.0.3, decord, scikit-image, scenedetect, pyloudnorm for audio normalisation, opencv-python, dashscope, loguru, ftfy, easydict and tqdm.

The checkpoints are separate downloads, and there are two of them. The SoulX-LiveAct model is on Hugging Face and on ModelScope, and the audio encoder is chinese-wav2vec2-base, which is on Hugging Face under the TencentGameMate organisation.

Four ways to run it, and the performance gap between them

The inference section gives four commands, and they differ in resolution, frame rate and editing support.

The first is the headline case, real-time streaming inference on two H100 or H200 GPUs, at 20 FPS and a 416 by 720 size, with a flag for printing the duration and one for Steam audio:

USE_CHANNELS_LAST_3D=1 CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node=2 --master_port=$(shuf -n 1 -i 10000-65535)  \
    generate.py \
    --size 416*720 \
    --ckpt_dir MODEL_PATH \
    --wav2vec_dir chinese-wav2vec2-base \
    --fps 20 \
    --dura_print \
    --input_json examples/example.json \
    --steam_audio

The second is the same two-GPU setup with action or emotion editing at real-time streaming performance, which uses a 512 by 512 size and 24 FPS and a different input file. The third is described as the best performance settings, at 480 by 832 and 24 FPS.

The fourth is the consumer GPU path, on an RTX 4090 or an RTX 5090, and it comes with a warning: FP8 KV cache may slightly affect generation quality. The command is a single-process python invocation rather than a torchrun across two devices, and the size drops to 416 by 720.

The performance figure for that path comes from the News section rather than the inference section. From 18 March 2026 the project supports consumer GPUs such as the RTX 4090 and RTX 5090 with FP8 KV cache and CPU model offloading, and in their tests the 18B model, which is 14B of Wan2.1 plus a 4B audio module, achieves a throughput of 6 FPS on a single RTX 5090. So the gap between the published 20 FPS on two H100s and the 6 FPS on one consumer card is roughly a factor of three from the hardware difference and a factor of two from the device count.

The USE_CHANNELS_LAST_3D environment variable appears in every command, which tells you it is not optional. And the master port is picked at random from a port range, which is the standard way to let several torchrun jobs share a host.

What is released, what is not, and the updates since the launch

The open-source plan is a checklist, and it is honest about the one item still open. Released are the inference code and checkpoints, GUI demo support, end-to-end adaptive FP8 precision, support for model offloading for consumer GPUs to reduce memory usage, and support for FP4 precision for B-series GPUs. Not released is the training code.

That gap matters for anyone who wants to retrain or fine-tune, and it is the difference between a system you can use and a system you can adapt.

The News section records the pace of work since the launch on 16 March 2026, when the inference code and model weights were released. On 18 March 2026 came the consumer GPU support with FP8 KV cache and CPU model offloading. On 11 May 2026 the model was updated and several bugs fixed to reduce excessive sharpening around the lips, teeth and other facial details, which is a very specific class of artefact for a talking-head model. On 27 May 2026 FP4 GEMM support was updated for B-series GPUs, including the RTX 5090, RTX PRO 6000, B100 and B200.

That sharpening note is worth reading as a signal about the output quality of a real-time talking-head system, and the FP4 and FP8 work is the signal about where the performance is coming from. The system is trading precision for speed, in a way that is well understood in the diffusion serving world but that a user of the demo will notice.

The repository layout is small and organised around the model. There is generate.py as the entry point, demo.py for the GUI, fp4_gemm.py and fp8_gemm.py for the two precision paths, model_liveact/ for the model code, wan/ and kokoro/ for the component models, src/ for the rest, util_liveact.py for utilities, templates/ and assets/ for the demo, and examples/ with the example.json and example_edit.json input files plus audio/ and image/ directories.

The last push was on 2026-06-15, the default branch is main, there are no GitHub releases, and the repository licence is not classified by GitHub.

Editorial conclusion

SoulX-LiveAct suits a research team that needs to reproduce or build on a real-time audio-driven human animation system, since the inference code and checkpoints are released and the three technical contributions are stated precisely enough to build on. It is the wrong choice if you need training code, because the open-source plan leaves that unchecked, and it is the wrong choice if you have one consumer GPU and expect the published performance, because the two-GPU number assumes an H100 or H200 and the single-GPU path is quoted at 6 FPS. Before running it, note that the install pulls three separate compiled dependencies at pinned versions, that FP8 KV cache may slightly affect generation quality on consumer cards, and that the model weights are separate downloads on Hugging Face and ModelScope.

Frequently asked questions

Official sources

1. Issues
2. README
3. Soul-AILab/SoulX-LiveAct on GitHub