AniGen: generating rigged 3D assets from a single image

The core idea, stated as three fields over one domain

AniGen is described as a unified framework that directly generates animate-ready 3D assets conditioned on a single image. The key insight the project claims is to represent shape, skeleton, and skinning as mutually consistent S3 Fields, for Shape, Skeleton, Skin, defined over a shared spatial domain.

That is the sentence to hold on to, because everything else in the repository follows from it. The problem with generating a mesh, a skeleton and skinning weights separately is that they can disagree. A mesh that has an arm where the skeleton has no bone, or skinning weights that blend across a joint that does not exist, produces an asset that looks plausible and animates badly. By putting all three in one spatial domain, the network is forced to make them consistent by construction rather than by a reconciliation step afterwards.

The Overview section restates the same idea in output terms. AniGen takes a single image as input and automatically produces a fully rigged, animate-ready 3D asset, complete with a coherent mesh, an articulated skeleton, and smooth skinning weights. The generated assets can be directly imported into standard 3D pipelines and driven by off-the-shelf motion data.

The stated downstream applications are the ones that make a rig generator interesting beyond game art. Embodied AI agent construction, physics-based simulation, character animation, dynamic scene creation, and articulated object manipulation. The gallery of examples reflects that mix rather than a single category, with a machine arm, a machine dog, a money tree, an iron boy, and then animals and humanoids, which is also the category list the abstract claims generalisation across, namely animals, humanoids, and machinery.

The project is written in Python, the default branch is main, and the last push to the repository was on 2026-07-15. The project page is at yihua7.github.io/AniGen_web/ and the paper is on arXiv at 2604.08746.

The two technical innovations, and why each one is needed

The abstract names two innovations, and the interesting part is what problem each one solves.

The first is a confidence-decaying skeleton field that explicitly handles the geometric ambiguity of bone prediction at Voronoi boundaries. A skeleton field assigns a bone to every point in space, which in practice means partitioning space, and the partition is what produces the ambiguity. Near a boundary between two bones, the network has to decide which bone a point belongs to, and at the boundary itself there is no right answer. A naive field produces a hard assignment, and a hard assignment produces a discontinuity, and a discontinuity in the skinning weights produces a visible seam when the joint bends. A confidence that decays with distance to the boundary gives the network a way to express uncertainty instead of forcing a decision.

The second is a dual skin feature field that decouples skinning weights from specific joint counts, allowing a fixed-architecture network to predict rigs of arbitrary complexity. This one is a direct answer to a training constraint. A network that predicts one skinning weight per joint has a fixed output dimension, so it can only handle rigs with a fixed number of bones. Real assets vary, from a simple machine arm to a human hand. Decoupling the skinning representation from the joint count lets the same network produce a rig of any size, which is what makes the single-image setting work at all across categories.

Both innovations are in the same category, which is worth naming: they are both ways of representing ambiguity rather than resolving it. The confidence decay and the dual feature are both about letting the model say not quite sure in a structured way, rather than committing to a discretisation that the training data does not actually support.

Two stages, a sparse scaffold and a dense result

The pipeline is described as a two-stage flow-matching pipeline. AniGen first synthesizes a sparse structural scaffold and then generates dense geometry and articulation in a structured latent space.

The ordering is what makes the approach tractable. A scaffold is the small, hard part: a handful of bones, their connectivity, roughly where they sit. Dense geometry and skinning is the large, smooth part. Generating the sparse structure first and conditioning the dense generation on it means the hard discrete decision is made once, in a space where the answer is small, and the expensive per-point prediction is conditioned on an answer that is already consistent.

The repository layout matches that structure. There is an anigen/ directory holding the package itself, a configs/ directory, an extensions/ directory for the compiled parts, an assets/ directory with the teaser image and the animation gifs, a train.py and an example.py at the root, a setup.sh, a requirements.txt, an app.py, and a video_editing/ directory.

The requirements.txt is worth reading for a different reason, because its header explains what is missing and why.

# AniGen core dependencies (Python 3.10+)
#
# NOTE: torch, torchvision, and compiled CUDA extensions (pytorch3d, nvdiffrast,
# spconv) are NOT listed here — they require CUDA-version-specific wheels.
# Install them via:  . ./setup.sh --all
# Or see README.md for manual installation instructions.

The four omissions are torch, torchvision, pytorch3d, nvdiffrast and spconv, and the reason given is that they need wheels matched to the local CUDA version. That is a real constraint rather than an omission, because pytorch3d, nvdiffrast and spconv are compiled against a specific CUDA and PyTorch combination, so a pip install of a pinned version would fail on a machine with a different CUDA. The header therefore points at the setup script instead.

The rest of the dependency list is a normal research stack, with numpy held below 2.0, onnxruntime, trimesh and pygltflib for geometry, xatlas for parameterisation, pymeshfix and rembg for mesh cleanup and background removal, pyvista for visualisation, geffnet and transformers for the network backbones, safetensors for checkpoints, and utils3d pinned to a specific commit hash from a GitHub fork rather than a released version.

Fitting a rig to a video, with differentiable rendering

The first downstream application is monocular 4D reconstruction with generative priors. The idea is to fit a generated rig to a monocular video and animate it, and the README says the skeleton motion is optimized by differentiable rendering, guided by LoMa appearance correspondences between the rendered frame and the masked frame, followed by a joint pose plus skin plus texture refinement.

There is one command for it, and the invocation is dense enough to be worth reading closely.

bash anigen/4drecon/run.sh --video clip.mp4                # or:  --frames frames_dir/
                           [--mask mask.mp4 | masks_dir/]  # optional; if omitted, SAM3 auto-masks
                           [--prompt "robot"] [--name myseq]

The input is either a video or a folder of frames, and the mask is optional. The comment explains what happens without it, the masks are produced automatically by SAM3 using the `--prompt` text. So a subject segmentation model is in the loop, driven by a text prompt, which is why the command has a `--prompt` argument at all.

The README also states the boundaries of this feature clearly. It is an optional add-on under anigen/4drecon/, and its dependencies and third-party checkouts, VGGT-Omega and LoMa, are installed separately, with a pointer to that directory's own README.

The rows of example images show what the output contains for each input, with the caption stating that each row is the input video, then the mesh overlay, then the skeleton overlay, then the AniGen prediction. The overlays are there to make the intermediate steps legible, which is a reasonable choice for a method whose contribution is the rig, not the mesh.

Video editing, and a warning that it is a preview

The second application is video editing, where a fitted asset is given a new motion and a new camera. The rendering is done by VACE, identified in the README as Wan2.2-VACE-Fun-A14B.

The command takes the asset, the reconstruction directory, the mask directory and the two new motion sources.

cd video_editing
python edit.py --asset unitree_g1_1 --recon <recon_dir> --mask <recon_dir>/dynamic_mask --name jumpingjack \
    --motion_source "do two jumping jacks then wave" --camera_source "slowly orbit the robot"

The recon_dir is described as a one-off scene reconstruction of the source clip, meaning depth plus cameras plus masks, and the mask argument is the per-frame subject masks, which live in the recon's dynamic_mask/ directory. The full guide is in the video_editing README.

The important thing in this section is the warning, and it is a good one. The note says the current video editing tool is a preview version, and that the authors are developing another video editing research that supports non-rigid motion editing, 6DoF trajectory editing, scaling, duplication, deletion, and camera re-shooting.

Two things follow. The tool as it exists does motion and camera replacement on an already-fitted asset, and it is explicitly not the final version. The separate mention of Tripo in the badge row at the top, linking to tripo3d.ai as a 3D workspace, suggests the intended workflow involves an external platform for the parts this repository does not cover.

The scripted nature of the inputs is also worth noting. The motion source is a text prompt like do two jumping jacks then wave, and the camera source is a prompt like slowly orbit the robot. The generation of a new camera trajectory from a text description is the part that the announcement flags as not yet supported, and it is the part that would turn a repurpose tool into a general editing tool.

Licence, releases and what a consumer inherits

Three things in the repository are worth calling out for anyone considering it, and none of them are in the abstract.

The first is the licence. The repository's licence field records a custom licence that GitHub cannot classify, and there is a LICENSE file at the root and a separate THIRD_PARTY_LICENSES.md file, which is the file you would expect in a repository that vendors third-party code, and this one vendors checkouts of VGGT-Omega and LoMa for the 4D reconstruction path as well as a pinned fork of utils3d in its requirements. A research codebase with a custom licence and a third-party licence inventory is workable, but the terms need to be read before any commercial use, and the README does not summarise them.

The second is that the repository publishes no GitHub releases. There is no version tag to pin to, so a checkout of main is the artefact, and a checkout of main is by definition a moving target. For a research codebase that is normal, and for a production dependency it is disqualifying.

The third is the install story. The requirements.txt header is explicit that torch, torchvision and the compiled CUDA extensions are not there because they need CUDA-version-specific wheels, and points at setup.sh with the --all flag, or at manual instructions in the README. That is honest and correct engineering, but it means the difference between a working checkout and a broken one is a script that installs the right binary for your CUDA, and the environment needs a GPU to be worth setting up at all.

The repository also carries AGENTS.md and CLAUDE.md at the top level, which is the convention for coding agent instructions in a repository, alongside the .gitignore. Those files are the closest thing to a contributor guide, and the setup.sh script is the closest thing to an installation guide, since the README here points at an earlier section that has been cut off.

Where AniGen is the wrong choice

Three cases point elsewhere. The first is a production asset pipeline. This is a research system, published alongside a SIGGRAPH paper, with no releases, a custom licence and a custom compiled dependency chain. A studio that needs to generate a rigged asset today with a predictable licence, a reproducible install and a support contact needs something else.

The second is a machine without a recent GPU. The compiled extensions named in the requirements header, pytorch3d, nvdiffrast and spconv, exist because the differentiable rendering and the sparse convolutions need them, and they are the parts that pin the install to a specific CUDA version. The README describes no CPU path anywhere.

The third is a project that needs precise control over the rig rather than a plausible one. The generated skeleton is a prediction, and the abstract's own claim is about rig validity and animation quality against sequential baselines, not about matching a specific topology. A rigging tool that takes control of a skeleton, an artist's clean bone hierarchy or a production rig from a DCC application is solving a different problem, and the 4D reconstruction path only goes in the direction of fitting the generated rig to video rather than editing an existing one.

The alternative worth weighing is a rigging tool that operates on an existing mesh, a rigging model in a graphics package, or a two-stage approach of generating a mesh and rigging it separately with an established auto-rigging system. The difference in approach is exactly what the S3 Fields formulation is trying to change. Generating the mesh, the skeleton and the weights independently and reconciling them afterwards is a pipeline with a repair stage, and the repair stage is where the visible artefacts come from. AniGen's claim is that making the three mutually consistent in a shared domain removes the need for that stage, and the confidence-decaying field and the dual skin feature are the two specific mechanisms for handling the cases where the shared domain is genuinely ambiguous. If that claim holds on your categories, the output is a rigged asset rather than a mesh plus a repair problem, and the cost is that you have to run a research system with its own dependency stack to get it.

Editorial conclusion

AniGen fits a research group in embodied AI, simulation or animation that needs rigged assets from images and is prepared to work from a research codebase with custom compiled dependencies. It does not fit a production pipeline that needs a stable API, a permissive licence or a reproducible install on arbitrary hardware, because the licence is a custom one, the repository publishes no releases and the CUDA extensions have to be installed per CUDA version. Before you build on it, read LICENSE and THIRD_PARTY_LICENSES.md, and run setup.sh rather than pip installing the requirements, since the compiled extensions are deliberately excluded from requirements.txt.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. VAST-AI-Research/AniGen on GitHub