Lift4D: three stages, three gated checkpoints, one A100

Stage one only runs for videos you brought yourself

The pipeline is three stages ending in a 4D asset: segment the object with a text prompt using SAM 3, reconstruct one 3D object per frame with Causal SAM3D using causal latent propagation, then fuse those per-frame reconstructions into a canonical shape plus a learned deformation. The selection that trips people up is that stage 1 is not universal. Three dataset types are read, `davis`, `consistent4d` and `custom`, chosen with `--dataset` at every stage. DAVIS and Consistent4D ship with masks already, so they skip segmentation entirely. Only a `custom` clip of your own goes through the segmenter, which writes masks to `data/custom/<video>/masks/`:

python segment_video.py \
    --video data/custom/goat.mp4 \
    --frames_dir data/custom/goat/frames \
    --output_dir data/custom/goat/masks \
    --prompt "goat"

The same script works from frames you already extracted by omitting `--video`, which matters when your video tool has already done the decode. One object per clip is the unit of work throughout, so a scene with two moving subjects means two runs and two assets.

Three checkpoints, two of them licence-gated

Weights come from three separate Hugging Face repositories, and the first line of the block is the one that stops people:

hf auth login

hf download --repo-type model --local-dir sam3d/checkpoints/hf-dl facebook/sam-3d-objects
mv sam3d/checkpoints/hf-dl/checkpoints sam3d/checkpoints/hf && rm -rf sam3d/checkpoints/hf-dl

# SAM3 ->  sam3/checkpoints/sam3/
hf download facebook/sam3 sam3.pt config.json --local-dir sam3/checkpoints/sam3

# Stable Zero123  ->  lift4d_scgs/extern/ldm_zero123/stable_zero123.ckpt
hf download --repo-type model stabilityai/stable-zero123 stable_zero123.ckpt \
  --local-dir lift4d_scgs/extern/ldm_zero123

Two things to note. SAM3D and SAM3 are gated, so `hf auth login` only works after you have accepted the upstream licence in a browser, and an unauthenticated `hf download` fails rather than falling back. Second, the third repository is Stable Zero123, downloaded into `lift4d_scgs/extern/ldm_zero123`. A single-view estimator sitting under the stage that is supposed to harmonize single-view estimates is inherited from the SCGS base rather than added for the harmonization, which tells you something about what stage 3 is building on.

Kaolin is pinned to torch 2.5.1 on cu121

The install is one conda environment, created from a file inside the SAM3D subdirectory and then shared by all three stages, which is a neat trick that also means the whole pipeline inherits one torch version:

git clone https://github.com/yehonathanlitman/Lift4D && cd Lift4D

#  Create the conda env `lift4d` (one env, shared by all three stages)
conda env create -f sam3d/environments/default.yml
conda activate lift4d

cd sam3d
export PIP_EXTRA_INDEX_URL="https://pypi.ngc.nvidia.com https://download.pytorch.org/whl/cu121"
pip install -e .
pip install -e '.[p3d]'
export PIP_FIND_LINKS="https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.5.1_cu121.html"
pip install hatchling hatch-requirements-txt ninja editables
pip install -r requirements.inference.txt --no-build-isolation
./patching/hyd

The `PIP_FIND_LINKS` line is the fragile one, since it points at a bucket of Kaolin wheels built for torch 2.5.1 on CUDA 12.1. That fixes your torch and CUDA generation before you have chosen a driver, and `--no-build-isolation` means the build needs `hatchling`, `hatch-requirements-txt`, `ninja` and `editables` already present, which is why the manual install of those four precedes the requirements file. The last step runs a patch script rather than a pip command. The only hardware the documentation names is a single A100 40GB GPU.

The mask filename is the contract between stage 1 and stage 2

Masks come out of segmentation as `frame_XXXXXX_<prompt>_<instance>.png`, so the thing stage 2 needs is not a path but the trailing `<prompt>_<instance>` pair, given as `goat_1` in the example. Stage 2 consumes it as `--mask_name`:

cd sam3d
# DAVIS
python run_inference.py --dataset davis --video_consistency 0.2 --object_name rhino --render_video
# consistent4d
python run_inference.py --dataset consistent4d --video_consistency 0.1 --object_name scaring_skull --render_video
# custom (mask_name = <prompt>_<instance> from stage 1, e.g. goat_1)
python run_inference.py --dataset custom --object_name goat --mask_name goat_1 \
     --video_consistency 0.2 --render_video --num_frames 64 #--subsampling 4

The handoff is stringly typed. Your text prompt becomes part of an identifier another stage has to reproduce exactly, so a prompt with a space, a different spelling, or an instance numbering that does not match leaves stage 2 with nothing to load, and the error will point at a missing mask rather than at the prompt you typed. Read the instance number off the actual filenames on disk after segmenting instead of assuming it is 1.

One consistency scalar covers the whole sequence

Stage 2 has four documented knobs. `--subsampling N` keeps every Nth frame, and `--num_frames N` caps how many frames get reconstructed after subsampling, defaulting to 100 with 0 meaning all of them. `--render_video` writes a side-by-side `comparison.mp4`. The one that carries judgement is `--video_consistency`, a value in `[0,1]` where higher preserves more structure, which the documentation suggests for rigid objects. The examples use 0.2 for DAVIS, 0.1 for Consistent4D and 0.2 for the custom clip, and no guidance is given for choosing between them, so the setting is a per-sequence decision you make by watching `comparison.mp4` and adjusting. Two things run by default that most people discover only by reading the options: Depth Anything 3 is executed over the selected frames unless you pass `--no-run_da3`, a double negative worth knowing about before you go looking for a speed flag that does not exist.

Pose repair is off, and bad alignment survives into the asset

The documented diagnostic is visual. If pose alignment with the object in the video is bad in `comparison.mp4`, the fix is `--refine_pose`, which optimizes each frame's pose so its rendered mask matches that frame's input mask. The documentation states it will improve final appearance quality when the initial alignment is bad, and that it is off by default. That default is the right call for cost and the wrong one for a first run, because stage 3 fuses whatever per-frame reconstructions it is handed, so an object floating slightly off its own silhouette in stage 2 becomes a canonical shape fitted to a bad set of poses rather than a correctable downstream artefact. Read `comparison.mp4` before training, and treat a visibly misaligned frame as a reason to re-run stage 2 with refinement rather than as a cosmetic issue to live with.

Consistent4D arrives through a personal Drive link

Dataset acquisition is uneven in quality. DAVIS comes from an institutional mirror over plain wget, while Consistent4D is a Google Drive link unpacked with `python -m zipfile -e data/CONSISTENT4D_DATA.zip data/consistent4d/`, with no checksum given for either archive and no stated policy for what happens when that Drive link moves. Custom clips are meant to be ordinary footage, with a Pexels video offered as the worked example and a note to take the 960x540 file. Paths are configurable through `LIFT4D_DATA_ROOT`, `CUSTOM_ROOT`, `CONSISTENT4D_ROOT` and `DAVIS_ROOT`, or through a `--data_root` flag, which is the one piece of genuine flexibility in the layout. Stage 3 then runs from `lift4d_scgs` with a shared argument string, for example `DS="--dataset consistent4d --object_name scaring_skull"`, applied to both the geometry and appearance steps. The visible documentation stops partway through that first step. The last push to main was on 2026-08-02 and there are no GitHub releases, so there is no tagged version to record against a result.

Editorial conclusion

Adopt Lift4D if you are doing 4D reconstruction research on a single A100-class GPU and you already have a Hugging Face account willing to accept the SAM3 and SAM3D licences. Do not adopt it for production throughput or for multi-object scenes, since the documentation covers one object per clip and the final visible stage step is cut off mid-instruction. Verify first that `comparison.mp4` from stage 2 shows your object sitting correctly in the frame, because the pose repair that fixes bad alignment is off by default and the finished asset inherits whatever stage 2 got wrong.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. yehonathanlitman/Lift4D on GitHub