JoyVASA: Audio-Driven Portrait and Animal Animation via Decoupled 3D Keypoints
Diffusion-based Portrait and Animal Animation
At a glance
- What is it?
- JoyVASA splits facial animation into a static 3D appearance representation and a diffusion-generated motion sequence, which lets it animate animal faces and stitch longer videos. The trade-off is a multi-repository checkpoint setup and an inference pipeline that inherits LivePortrait's rendering stack.
- Who is it for?
- Adopt JoyVASA if you need audio-driven animation that works on both human portraits and animal faces, and you can accept a heavy checkpoint setup: a JoyVASA motion generator, a LivePortrait rendering stack, insightface detectors, and either wav2vec2-base or chinese-hubert-base as the audio encoder. Skip it if you need real-time output, since the README states real-time performance is future work, or if you want a single pip install with no external model download.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 153 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What JoyVASA Actually Generates and for Whom
JoyVASA takes a still reference image and an audio clip, and produces a video where the face in the image speaks and moves its head in time with the audio. The repository describes it as a diffusion-based method for generating facial dynamics and head motion in audio-driven facial animation. Two output channels are named: facial expression dynamics and head motion. Lip-sync is implied by the audio-driven framing and by the audio-driven-talking-face topic tag, though the README does not give a separate lip-sync metric or evaluation table.
The intended users are teams building talking-head or image-animation features who need multilingual support and, unusually, non-human subjects. The README states the model was trained on a hybrid dataset of private Chinese and public English data, and that the decoupled representation and identity-independent motion generation let JoyVASA extend beyond human portraits to animate animal faces. That animal case is the differentiating claim. Most audio-driven portrait systems in this space assume a human face detector and a human 3D prior; JoyVASA's motion stage is trained to be identity-independent, so the same motion generator can drive a non-human reference image.
It is not a product. There is no hosted demo link that is active in the README (the Hugging Face Space badge is commented out). You run it locally, on your own GPU, against your own images and audio.
The Two-Stage Decoupling and the Inference Data Flow
The architecture has two stages, and the split is the whole point.
Stage one is a decoupled facial representation. The README describes it as separating dynamic facial expressions from static 3D facial representations. A static 3D facial representation is extracted once per reference image and can then be combined with any dynamic motion sequence. That is what allows longer videos: you are not re-encoding the identity for every frame, and you can concatenate motion segments without the identity drifting.
Stage two is a diffusion transformer trained to generate motion sequences directly from audio cues, independent of character identity. Because the motion model never sees the identity, it generalizes across subjects, including animals.
The inference pipeline, as described in the README, runs in this order. A reference image goes through the appearance encoder in LivePortrait to produce a 3D facial appearance feature. A motion encoder produces a series of learned 3D keypoints from the same image. Separately, the input speech is encoded by wav2vec2. The diffusion model from stage two samples audio-driven motion sequences in a sliding window fashion. Those sampled target motion sequences are combined with the reference image's 3D keypoints to compute target keypoints. Finally the 3D facial appearance feature is warped based on source and target keypoints and rendered by a generator into the output video.
The sliding window is the mechanism that bounds memory: the diffusion model never has to denoise an entire utterance at once. The cost is that continuity across window boundaries is a stitching problem, not something the model solves globally. The README does not describe how boundary frames are blended, so if you care about long-form output, that is worth inspecting in the code before you build on it.
Note the dependency direction: JoyVASA does not reimplement rendering. It uses LivePortrait's appearance encoder, motion extractor and generator, and adds its own audio-conditioned motion generator on top.
Getting It Running: Environment, Checkpoints and the Directory Layout
The README gives an explicit setup sequence. Create the environment with conda create -n joyvasa python=3.10 -y, activate it, then pip install -r requirements.txt. ffmpeg is a hard requirement: sudo apt-get update followed by sudo apt-get install ffmpeg -y.
Animal animation needs one extra build step. The README marks it optional and points at src/utils/dependencies/XPose/models/UniPose/ops, where you run python setup.py build install. That is MultiScaleDeformableAttention, and it is a compiled CUDA extension, so it will fail on a machine without a matching toolchain even though the rest of the install succeeds.
Checkpoints are the real work. Everything lands in pretrained_weights. The motion generator and motion template come from git clone https://huggingface.co/jdh-algo/JoyVASA after git lfs install. The audio encoder is one of two: git clone https://huggingface.co/TencentGameMate/chinese-hubert-base, or git clone https://huggingface.co/facebook/wav2vec2-base-960h. The LivePortrait weights come from huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs".
The expected tree is specific, and mismatches here are the most likely first failure. insightface/models/buffalo_l holds 2d106det.onnx and det_10g.onnx. JoyVASA/motion_generator holds iter_0020000.pt, and JoyVASA/motion_template holds motion_template.pkl. liveportrait/base_models holds appearance_feature_extractor.pth, motion_extractor.pth and spade_generator.pth. If you clone a repo into the wrong subdirectory, the loader will look for a path that does not exist.
One caveat the README states directly: the motion generation model with the wav2vec2 encoder will be supported later. So although wav2vec2-base is listed as a supported audio encoder, the released motion generator is paired with chinese-hubert-base. If you are working in English, verify this against the current code before assuming parity.
Where JoyVASA Breaks Down or Is the Wrong Choice
Real-time is explicitly out of scope for this release. The README's own future-work sentence names improving real-time performance and refining expression control. A diffusion transformer sampled in a sliding window, feeding a warping and rendering stage, is not a streaming architecture. If your use case is a live avatar or a low-latency call, this is the wrong tool today.
Expression control is the second gap, and it is admitted in the same sentence. There is no documented interface in the README for specifying emotion, blink rate, or gaze. You supply audio and an image; the model decides the rest. If your product needs an animator to override a head turn or damp an expression, nothing in the supplied material shows how.
Hardware is a real constraint. The README lists tested configurations, not minimums: Ubuntu 20.04 with CUDA 12.1 on an A100, and Windows 11 with CUDA 12.1 on an RTX 4060 Laptop with 8GB VRAM. Those two data points span a wide range, and the README does not state which stages ran on which. The 8GB laptop result suggests the pipeline can fit in consumer memory, but the note is thin and the tested-GPU list is not a guarantee for your card.
The checkpoint sprawl is a maintenance liability in itself. You are pulling from three separate Hugging Face repositories plus an insightface model directory, under two different licences at least (JoyVASA is MIT; the LivePortrait and insightface components carry their own terms, which the README does not restate). Nothing in the material describes a single-command setup or a Docker image.
Finally, the training data is described as a hybrid of private Chinese and public English data. Public English data means the English behaviour is the part you can reason about from outside; the Chinese portion is not inspectable. That is a reproducibility boundary, not a defect, but it limits how much you can predict about edge cases.
How It Differs from LivePortrait and Other Talking-Head Pipelines
The most direct comparison is LivePortrait, because JoyVASA is built on top of it and the README says so. LivePortrait is a video-driven portrait animation system: you give it a driving video and a source image, and it transfers the motion. JoyVASA replaces the driving video with audio. It keeps LivePortrait's appearance encoder, motion extractor and spade_generator for the rendering half, and inserts a diffusion transformer that predicts 3D keypoint motion from wav2vec2 or hubert features. So the difference is not quality tuning; it is a different conditioning signal, with the entire motion source swapped from a reference video to a speech waveform.
That swap has a consequence worth naming. Video-driven methods inherit the driving video's head motion, so the output looks natural by construction. Audio-driven methods have to invent head motion, and that is exactly the part JoyVASA trains separately and identity-independently. It is also the part most likely to look generic or repetitive, and the README offers no ablation showing how much of the perceived naturalness comes from the motion generator versus the renderer.
Against end-to-end audio-driven talking-head models, the architectural difference is the decoupling. An end-to-end model maps audio to pixels directly, which couples identity and motion in one network and makes long videos harder because errors accumulate per frame. JoyVASA's stage one separates the static 3D representation from the dynamic sequence, which is what the README credits for longer videos and for animal faces. The cost is a longer pipeline with more places for a checkpoint path or a coordinate convention to go wrong.
If your requirement is a single self-contained model with one weight file, JoyVASA is more machinery than you need. If your requirement is long-form output or non-human subjects, the decoupling is the reason to pick it over a monolithic alternative.
Licence and the Cost of Keeping It Current
JoyVASA itself is MIT, which is permissive for commercial use of the code. That does not cover everything you download. The pipeline pulls LivePortrait weights from KwaiVGI/LivePortrait, insightface buffalo_l detectors, and an audio encoder from either facebook or TencentGameMate. Each of those has its own licence and its own terms, and the README does not summarise them. Check each one against your distribution model before shipping; this is not legal advice, just a note that MIT on the top-level repository is not the whole picture.
The repository has no tagged releases. The README's release history is empty, so there is no version number to pin and no changelog to read. Upgrades mean pulling main and re-checking that the pretrained_weights layout still matches the tree the README documents. The motion generator is a single file, iter_0020000.pt, and the motion template is motion_template.pkl; if a future commit changes the expected filenames or the template format, your existing checkpoints may stop loading. Budget for that by keeping a known-good copy of pretrained_weights rather than re-cloning on every update.
The upstream dependency on LivePortrait is the other ongoing cost. JoyVASA consumes LivePortrait's base models, so a change in how LivePortrait packages or names those files ripples into JoyVASA's loader. The README already hedges on one encoder pairing, stating that wav2vec2 support will come later, which suggests the supported matrix is still moving. Treat the documented configuration as the only configuration, and test any deviation yourself.
Editorial conclusion
Adopt JoyVASA if you need audio-driven animation that works on both human portraits and animal faces, and you can accept a heavy checkpoint setup: a JoyVASA motion generator, a LivePortrait rendering stack, insightface detectors, and either wav2vec2-base or chinese-hubert-base as the audio encoder. Skip it if you need real-time output, since the README states real-time performance is future work, or if you want a single pip install with no external model download. Before committing, verify three things: that your GPU has enough VRAM to hold the LivePortrait generator alongside the diffusion motion model, that your target language matches the audio encoder you pick (chinese-hubert-base for Chinese, wav2vec2-base for English), and that the MultiScaleDeformableAttention op builds on your CUDA version if you intend to animate animals.
Community notes