Face Anything: a feed-forward model that reconstructs and tracks faces in 4D from any image sequence

Reconstruction and tracking as one canonical problem

Face Anything is a unified feed-forward model for high-fidelity 4D face reconstruction and dense tracking from arbitrary image sequences. The README's key idea is canonical facial point prediction: a representation that assigns each pixel a normalized facial coordinate in a shared canonical space, which transforms dense tracking and dynamic reconstruction into a single canonical reconstruction problem, producing temporally consistent geometry and reliable correspondences.

The user is a researcher or engineer in face capture, avatars or vision who needs both the 3D shape over time and the point-to-point correspondences across frames, from ordinary image sequences rather than a controlled capture rig. Being feed-forward means it predicts in a single forward pass rather than optimizing per sequence, which is a speed and simplicity advantage over fitting-based methods.

The unifying insight is the distinguishing contribution. Treating tracking and reconstruction as one canonical problem, rather than two pipelines stitched together, is what yields temporally consistent geometry and correspondences at once. It ships a Hugging Face Space demo, and its license is reported as NOASSERTION, which is the first thing to check before use.

What a run produces

Face Anything's output is unusually concrete, which makes it easy to judge. The README describes a full run writing a set of videos and data: a colored 3D point-cloud reconstruction with an orbiting camera, a reconstruction with temporally-consistent point tracks, one colored by canonical facial coordinates, ones colored by depth and by surface normals, and a grand tour that morphs through every modality in one orbit, plus 2D image-space versions.

Beyond videos, it writes per-timestamp colored point clouds as `.ply` files for geometry, canonical coordinates and tracks, per-frame camera intrinsics and extrinsics in `cameras.npz` and `cameras.json`, and raw predictions, depth, intrinsics, extrinsics, canonical coordinates, confidence and validity, in `raw_predictions.npz`.

That output set tells you the model is meant to be a source of usable geometry and correspondences, not just a visualization. The presence of raw predictions with confidence and validity masks, and per-frame cameras, means the results can feed downstream pipelines rather than only being viewed. The canonical-coordinate output is the representation the whole method is built on, exposed directly so you can use the correspondences it encodes.

Installing and getting the checkpoint

Face Anything requires a CUDA GPU, and the README says it was tested with Python 3.11 and PyTorch 2.9 on CUDA 12.8. There is a one-line setup that creates the environment, installs everything and downloads the checkpoint:

git clone https://github.com/kocasariumut/FaceAnything.git
cd FaceAnything

bash install.sh

For a manual setup, the README uses a fresh conda environment and installs PyTorch matching your CUDA before the rest:

git clone https://github.com/kocasariumut/FaceAnything.git
cd FaceAnything

conda create -n faceanything python=3.11 -y
conda activate faceanything

The checkpoint is fetched separately with gdown, which the README shows explicitly:

pip install gdown
gdown --fuzzy "https://drive.google.com/file/d/1PdQQxzm-tU50RmJhgeoMCYVRlEiW3f8p/view?usp=sharing" \
    -O checkpoints/checkpoint.pt

The hard CUDA requirement and the specific PyTorch 2.9 / CUDA 12.8 test configuration mean a compatible NVIDIA GPU and a matching toolchain are prerequisites, not conveniences. For a quick look without any local setup, the Hugging Face Space demo runs the model in the browser.

The limitations: GPU-bound, domain-specific, unclear license

The honest limitations begin with hardware. Face Anything needs a CUDA GPU, tested on a specific PyTorch and CUDA pairing, so it is not something you run on a CPU or a non-NVIDIA machine. That bounds who can use it locally, though the Hugging Face Space lowers the barrier for trying it.

The method is also domain-specific by design: it is a face model built on canonical facial coordinates, so it applies to faces, not general scenes or objects. That specialization is the source of its quality, the canonical facial space is what makes the correspondences reliable, but it is not a general 4D reconstruction tool.

As a feed-forward model, its accuracy is bounded by its training distribution: it predicts in one pass, which is fast, but inputs far from what it was trained on, unusual imaging, extreme poses or occlusion, are where any feed-forward model degrades, and the raw-prediction confidence and validity outputs exist precisely because not every pixel's prediction is equally reliable. The license is the remaining caveat: GitHub reports NOASSERTION, so the terms are not cleanly resolved, and the separately-downloaded checkpoint may carry its own conditions, both of which a user should confirm before building on it.

Against per-sequence fitting or a landmark tracker

The alternatives are a per-sequence optimization method that fits a face model to each clip, or a sparse facial-landmark tracker. Per-sequence fitting can be very accurate but is slow, optimizing for each new sequence, whereas Face Anything predicts feed-forward in one pass. A landmark tracker gives sparse keypoints cheaply but not the dense geometry and dense correspondences Face Anything produces.

Face Anything's difference is delivering dense, temporally-consistent 4D geometry and dense correspondences in a single feed-forward pass, unifying tracking and reconstruction through the canonical representation. The cost is the CUDA requirement, the face-only domain, and the feed-forward accuracy ceiling. Choose per-sequence fitting when you need maximum accuracy on a few clips and can afford the optimization time. Choose a landmark tracker when sparse keypoints are all you need. Choose Face Anything when you want dense face geometry and reliable per-pixel correspondences over time from arbitrary image sequences, quickly, and you have the GPU to run it and have resolved the license terms.

An unresolved license, a demo, and where to start

The license question is the thing to settle before any serious use. Because GitHub reports NOASSERTION and the checkpoint is downloaded separately, treat the code's terms and the checkpoint's terms as distinct questions and confirm both, especially for anything commercial. Its last push was on 2026-08-05, so the code is recent.

Maintenance is research-style, paired with a Hugging Face Space and the usual paper artifacts. The dependency reality is a specific PyTorch and CUDA pairing plus the gdown-fetched checkpoint, so a working setup is that stack, and the `install.sh` one-liner is the path most people will take.

The concrete first step is to try the Hugging Face Space on your kind of footage before committing to a local install, since it lets you judge the reconstruction and tracking quality on real sequences without setting up CUDA. If the results fit, run `bash install.sh` on a CUDA machine with the tested Python 3.11 and PyTorch 2.9 / CUDA 12.8, produce a grand-tour run on your own sequence, and inspect the `raw_predictions.npz` confidence and validity to see how reliable the per-pixel geometry is on your data before trusting it downstream.

Editorial conclusion

Use Face Anything when you need dense, temporally-consistent 4D face geometry and reliable per-pixel correspondences from arbitrary image sequences in a single feed-forward pass: the canonical facial point prediction that unifies tracking and reconstruction, plus the rich output set of point clouds, tracks, depth, normals and per-frame cameras, is its real contribution. It is the wrong tool for non-face scenes, for machines without a CUDA GPU, or when you need per-sequence optimization-level accuracy. Resolve the license first, since GitHub reports the repo as NOASSERTION and the checkpoint is a separate download, then try the Hugging Face Space on your footage, and if it fits, run bash install.sh on a tested CUDA setup and check the raw_predictions confidence before trusting the geometry downstream.

Frequently asked questions

Official sources

1. Issues
2. kocasariumut/FaceAnything on GitHub
3. Project website
4. README