AI4Animation: Deep Learning Character Control for Unity and Python

What AI4Animation Is and Who It Serves

AI4Animation is a research repository that explores deep learning for character animation and control. The README describes it as aiming to be "a comprehensive framework for data-driven character animation, including data processing, neural network training and runtime control." Each sub-project demonstrates a specific neural network approach to animating characters: biped locomotion, quadruped locomotion, character-scene interactions, sports and fighting motion, and embodied avatar control in AR and VR contexts.

The primary audience is researchers working on motion synthesis and animation AI, technical animators at game studios experimenting with data-driven techniques, and developers who want to study how published neural animation methods work before deciding whether to implement them in a different context. The repository does not hide its research focus: every major project in it corresponds to a specific conference paper, primarily at SIGGRAPH.

Users looking for a ready-to-use animation plugin for a game engine are likely to be disappointed. AI4Animation provides the code and data pipelines behind specific research results. Using it productively requires understanding the paper it implements.

Repository Layout: Multiple Projects Under One Root

The top-level directory contains only three subdirectories: AI4Animation/, Media/, and two dot files. All of the actual project code is inside AI4Animation/, organized by conference year and paper. This means there is no single entry point to the repository; a reader has to identify which sub-project corresponds to the technique they want.

The Media/ directory contains figures, video thumbnails, and supporting visual material for the papers. The AI4Animation/ directory contains a subdirectory for each research project, such as SIGGRAPH_2024/ for the most recent entry. Each sub-project has its own ReadMe.md and links to its dataset and demo downloads.

The 2026 Python remake, AI4AnimationPy, is a separate repository at facebookresearch/ai4animationpy. The README mentions it at the top as a significant development that removes the Unity dependency and runs the entire pipeline on NumPy or PyTorch. The demos folder in that repository contains examples the README describes as directly runnable.

The Python Remake: AI4AnimationPy and What It Removes

The 2026 AI4AnimationPy remake is the most consequential change to the AI4Animation ecosystem described in the README. It eliminates the Unity dependency for data processing, feature extraction, inference, and post-processing. The README describes the architecture as keeping a game-engine-style design (ECS, update loops, rendering pipeline) while running entirely on NumPy or PyTorch.

The practical effect is that training, inference, and visualization can happen in a single Python environment without a Unity license or Unity Editor installation. The README lists the capabilities demonstrated in the Python remake's demos folder: a stylized biped locomotion controller trained on the style100 dataset, a quadruped locomotion controller with gait transitions, future motion anticipation with interactive training visualization, an ECS entity hierarchy, real-time inverse kinematics solving, motion capture import for GLB, FBX, BVH, and NPZ formats, and a motion editor for browsing animations.

A web demo is available at paulstarke-ai4animationpy.hf.space, which allows running some of the demos in a browser without any local installation. This is the fastest way to evaluate the Python remake's capabilities before cloning and setting up the local environment.

SIGGRAPH 2024: Categorical Codebook Matching for Embodied Avatars

The most recent research project in the repository is the SIGGRAPH 2024 paper on Categorical Codebook Matching for Embodied Character Controllers. The README describes the problem: translating motion from sparse sensor signals (three-point tracking from VR headset and controllers) to a full-body avatar motion in real time while preserving the motion context of the user.

The technical approach, called codebook matching, trains two categorical codebooks simultaneously, one for inputs and one for outputs, and enforces similarity between their probability distributions using a matching loss. The README describes the training and inference procedure:

Training:
\begin{cases}
    Y \rightarrow Z_Y \rightarrow Y
    \\
    X \rightarrow Z_X
    \\
    Z_X \sim Z_Y
\end{cases}

Inference:
X \rightarrow Z_X \rightarrow Y

During inference, the output codebook is replaced by the input codebook, removing the need for ground truth output signals at test time. The paper, dataset (Cranberry Dataset), VR demo, and Windows and Mac demos are all linked from the README. The code lives in AI4Animation/SIGGRAPH_2024/. The approach also supports a hybrid control mode where a joystick or button input adds a goal location to the three-point tracking signals.

Using the Repository: Datasets, Demos, and Setup

Each sub-project in AI4Animation requires its own dataset and has its own setup process. The README links to datasets for each paper. For SIGGRAPH 2024, the Cranberry Dataset is available at a direct download URL. Other sub-projects link to their own datasets.

For the Python remake, the recommended entry point is the demos folder in facebookresearch/ai4animationpy. The README states these demos are "supposed to give an idea what can be done with the framework" and can be run directly. For the original Unity-based projects, the setup requires Unity and the specific Unity packages listed in each sub-project's ReadMe.md.

There is no single install command for AI4Animation because the repository contains multiple independent projects. A developer interested in the biped locomotion technique from an earlier paper follows a different setup path than one interested in the SIGGRAPH 2024 embodied avatar work. Reading the sub-project's ReadMe.md is the correct starting point for each.

Limitations: Research Code, Not a Production Tool

AI4Animation is research code. Each sub-project was developed to validate the claims of a specific paper, and the code reflects that purpose. There is no unified API, no package manager entry, no versioned releases, and no support channel for production integration questions.

The Unity-based sub-projects require a compatible version of Unity, which changes over time. The README does not specify which Unity version is required for each sub-project, and Unity's API surface has changed significantly across versions. Older sub-projects may require version-specific fixes to compile.

The Python remake removes the Unity dependency but introduces its own: NumPy, PyTorch, and supporting libraries. The specific versions required for each demo are not listed in the excerpted README, which means environment compatibility needs to be tested rather than assumed.

The top-level repository metadata lists the primary language as C++ and the license as unknown. Individual sub-projects may have their own license files, but any commercial or redistributed use requires checking each sub-project's licensing terms individually. The last push to the main repository was on 2026-04-17.

AI4Animation vs Unity ML-Agents: Different Scopes

Unity ML-Agents is the official Unity package for training agents using reinforcement learning and imitation learning within the Unity environment. It is a general-purpose training framework with documentation, versioned releases, and active support from Unity Technologies.

AI4Animation is different in scope and intent. It is a research repository focused specifically on character animation using deep learning techniques, tied to specific papers. Where Unity ML-Agents provides a framework for training any kind of agent in Unity, AI4Animation provides reference implementations of specific motion synthesis architectures.

A developer using AI4Animation is essentially reading and running a published paper's code. A developer using Unity ML-Agents is adopting a supported tool for their own agent training work. The choice depends entirely on whether the goal is to reproduce or extend a specific animation research technique, or to build a new trained system from scratch using established infrastructure.

Editorial conclusion

AI4Animation is the right resource for researchers and technical animators who need a reference implementation of published neural character control techniques, particularly for projects presented at SIGGRAPH between 2017 and 2024. It is not a drop-in animation plugin for production games or applications. Anyone evaluating it should check the specific sub-project that matches their target technique, read the linked paper and dataset terms, and note that the Python remake at facebookresearch/ai4animationpy removes the Unity dependency for the data processing and inference pipeline. The repository has no attached license in the top-level metadata, so licensing terms for each sub-project need to be verified individually before any commercial or redistributed use.

Frequently asked questions

Official sources

1. Issues
2. README
3. sebastianstarke/AI4Animation on GitHub