GAE: putting the 3D inductive bias inside the generative latent space

The idea: geometry inside the state, not bolted on

GAE's central claim is stated in the README as putting the 3D inductive bias into the generated state itself. Instead of encoding a frame as an image and adding geometry from outside the latent, it compresses frozen geometry-foundation features into a compact per-view latent whose generated states decode jointly into RGB and geometry.

That is a design decision with real consequences, and the distinction matters. The common approach to 3D-consistent video generation is to generate images in pixel space and then reconstruct geometry from those images with a separate estimator, which means the geometry is only as consistent as the reconstruction step allows and errors accumulate. GAE inverts it: the geometry features are the state, and the RGB is a readout of the same latent.

The README's framing line for the project is camera-controlled video and 3D geometry, decoded from one shared latent. One latent, two decoders, so the video and the point cloud cannot disagree because there is nothing for them to disagree about.

This is a paper code release for the paper, with an arXiv identifier of 2609.24981 and a project page. The team is from The Hong Kong University of Science and Technology, the ARC Lab at Tencent IEG, the University of Hong Kong and the University of Texas at Austin, with Jiahao Lu and Minghao Yin as equal contributors and Wenbo Hu and Yuan Liu as corresponding authors.

The repository layout is organised around the two stages. The gae/ directory is for inference, scripts/demo/, scripts/train/, scripts/eval/ and scripts/data/ hold the CLIs, and configs/ holds the recipes. There are also app.py and camera_studio.py at the root, which are the interactive surfaces, an assets/ directory, model_stats/, examples/ and tests/.

The README is careful to say that internal research filenames are documented in docs/CODEBASE.md and docs/METHOD.md, and that you do not need them to run GAE. That is a courtesy to a reader who would otherwise assume the docs directory is mandatory reading.

The primary language is Python, the default branch is main, the last push was on 2026-10-13, and there are no GitHub releases. The licence is a custom one GitHub cannot classify, with a LICENSE.txt at the root, so the terms need reading before any commercial use.

Stage one: a codec sandwiched between two frozen ends

The pipeline diagram in the README is the clearest description of the architecture in the whole repository, and it is worth reading line by line.

images ──frozen DA3 encoder──> 4-level features ──level-wise normalize + concat──> X
   X ──GAECodec.encode──> z  (64 or 128 channels, DA3 patch grid)
   z ──GAECodec.decode──> rebuilt 4 levels ──frozen DPT head──> depth / rays / pointmap
   z ──learned RGB head──> RGB

Depth Anything 3, abbreviated DA3, is frozen at both ends. Its encoder produces a four-level feature hierarchy, and its DPT head reads depth, rays and point maps back out. GAE learns the state in between.

So the trainable part of stage one is a single autoencoder, GAECodec, sitting between a frozen encoder and a frozen decoder. The latent z is 64 or 128 channels on the DA3 patch grid.

The consequence of freezing the geometry head is stated bluntly in the README: because the geometry head never trains, geometry reconstruction has to stay readable by the model the features came from. That is a real constraint, not a soft one. It means the codec cannot drift into a latent space that is convenient for generation but that the frozen DPT head can no longer decode into coherent geometry.

That constraint is the whole reason for the loss terms, because reconstruction alone does not organise the bottleneck for generation.

L_tok aligns each projected posterior token with C-RADIOv2.5-B, which improves transport smoothness and semantic organisation. L_struct matches DINOv2-L patch-to-patch similarity directly in the raw posterior mean, restoring the spatial and cross-view structure that token-only alignment collapses.

Both terms act on the codec bottleneck before the flow model is trained, unlike REPA which aligns intermediate denoiser states during generative training. That is a meaningful architectural difference rather than a detail, and it is the kind of thing that separates a design choice in stage one from a training-time regulariser in stage two.

Stage two: a flow model over the frozen codec state

The second stage is where generation happens, and the README is precise about the arrangement.

The codec is frozen, its posterior mean is standardized per channel, and one conditional flow model is trained over that state with RAEv2-style x-prediction. Three controls specify the state without evolving with it.

The first control, clean reference latents, carries the subtlest reasoning in the project. It is described as evidence rather than state. DA3 is set-based, so a reference encoded alongside target views cannot be reproduced at inference. GAE therefore encodes observed references exactly as at inference and prepends them as clean tokens at t=0, and their outputs are discarded before the prediction head, with only generated views integrated.

That is a specific and clever arrangement. The alternative would be to condition on reference latents as an input to the model, but then the reference occupies a different role from the generated content, and the model has to learn to treat one as state and the other as evidence. Prepending reference tokens at t=0 and throwing their outputs away keeps a single state representation for everything, at the cost of spending some of the model's capacity on tokens whose output is thrown away.

The second control is metric Plücker ray maps, described as camera motion as a scale-aware control. Plücker coordinates describe a ray by two vectors, and the metric form makes the representation scale-aware rather than dependent on a chosen origin, which is what you want for camera control.

The third control is text, via cross-attention.

The payoff is stated in the README: the same backbone therefore covers text-to-image, camera-controlled video, and reference-conditioned novel-view synthesis. One trained model, three tasks.

The public weights are named GAE-D64-1B, which reflects the 64-channel latent and the one-billion parameter scale, and they are hosted on Hugging Face at TencentARC/GAE-D64-1B.

Installing it and running the demo

The install section is short and has one trap in it that is worth reading before you start.

python -m venv .venv && source .venv/bin/activate
pip install -e .            # installs the `gae` API and the src/ modules
# or, for a pinned CUDA env:  pip install -r requirements.txt

The trap is the venv creation. If python -m venv .venv fails with Operation not permitted on lib64, the checkout is on a filesystem that rejects the venv symlink, and the fix is to create the environment on local disk instead with python -m venv /tmp/gae-venv. run_demo.sh does that automatically, and the GAE_VENV variable overrides the location.

The Python version range is fixed by the torch build. The README says Python 3.10 to 3.12 and torch 2.5.1, and that python3.13 is not supported because pip install -e . will refuse it. The pyproject.toml encodes the same constraint with requires-python set to >=3.10,<3.13 and a comment noting that torch 2.5.1 wheels cover cp310 to cp312. The DA3-GIANT backbone is pulled from the Hub on first use, so the first run needs network access.

The editable install handles the import path for you: pip install -e . puts both gae and the research packages, stage1, stage2 and utils among others, on the import path, so no PYTHONPATH setup is needed. The scripts still add src/ themselves, so they also run from a plain requirements.txt checkout.

Inference is the fastest way to see the thing work, and the released weights turn one image plus a prompt into a multi-view video with a consistent 3D point cloud, or a prompt into a single image, using the same validated Euler and CFG sampler in both cases.

bash scripts/demo/run_demo.sh
bash scripts/demo/run_demo.sh --smoke          # 17 views, 25 steps
bash scripts/demo/run_demo.sh --task i2v

The smoke flag is the one to use first. It runs 17 views at 25 steps, which is a fraction of a full generation, and it is how you check the environment without waiting for a full run.

The script creates a virtual environment if needed, installs the package, downloads the weights when the ckpts/ directory is empty, and runs the bundled examples from examples/scenes/ plus the prompt file examples/t2i_prompts.txt. Results land in results/demo/i2v/<scene>/ and results/demo/t2i/.

The dependency surface, and what is optional

The dependency list is what tells you what this project actually needs at runtime, and it is worth reading for what is pinned versus what is ranged.

The hard pins are torch at 2.5.1, and the geometry foundation model depth-anything-3 at 0.1.0 or newer. Everything else is a range or unpinned: torchvision at 0.16.0 or newer, xformers, transformers, diffusers, accelerate, timm, safetensors and huggingface-hub, plus the config, maths and IO layer with omegaconf, einops, torchdiffeq, lpips, opencv-python, pillow, scipy, scikit-image, imageio, av, trimesh, plyfile, addict, e3nn, numpy, matplotlib, pyarrow, orjson and webdataset.

The optional extras in pyproject.toml are where the interesting engineering decisions are documented.

The eval extra holds evo and pycolmap, and the comment says it is only needed by the geometry evaluation entry points, scripts/eval/eval_geometry.py and scripts/eval/eval_3d_consistency.py, because they call external estimators. Those estimators are named: VGGT for the 3D consistency evaluation, MEt3R and Pi3 for the geometry evaluation. The comment also says torch_dct is optional too, because src/metrics/latent.py ships a pure-torch DCT fallback that gives identical results when it is absent.

A pure-torch fallback with identical results is a small thing that says a lot about how the project is written. It means a missing optional dependency degrades to a slower path rather than to a crash.

The muon extra is more interesting still, and the comment explains why it is not installed by default: it pins torch>=2.7.1 while this project sits on torch==2.5.1. The trainer falls back to AdamW when it is absent. So the optimiser choice is a genuine trade: the newer optimiser needs a newer torch than the project supports, and the project keeps the older torch and the older optimiser.

The data extra covers the OSP dataset loader and the BLIP3o webdataset path, adding orjson, pyarrow, webdataset and s3torchconnector. The requirements.txt also lists wandb and gradio, the latter for the Hugging Face Space UI, with the gradio pin being 6.15.1 or newer below 7.

Where GAE is the wrong tool

There are three limits, and the first is the one that decides everything for anyone who is not doing this research.

The first is that this is a paper release, not a library. The version is 0.1.0, there are no tagged releases, and the README describes it as the public code release for the paper. The directory naming even gives it away, with stage1, stage2 and utils as research packages alongside the gae inference API. Nothing in the README promises API stability, and the presence of the internal research filenames documented in docs/CODEBASE.md tells you the published surface and the working surface are not the same thing.

The second is the environment. Python 3.10 to 3.12, torch pinned to 2.5.1, a CUDA environment for the pinned requirements path, and a DA3-GIANT backbone pulled from the Hub on first use. The optimiser choice shows the fragility of that pin: the project cannot take a newer torch because one optional dependency wants torch>=2.7.1. On a machine where you need a current torch for other work, this repository is a constraint rather than a component.

The third is the licence. The repository metadata classifies it as Other, a custom licence GitHub cannot classify, and the pyproject.toml points at a LICENSE.txt file. For research use that is usually fine, but for anything commercial, the terms need to be read directly, because an unclassifiable licence is a signal that it is not one of the standard permissive ones.

The honest alternatives split by what you are trying to do. If you want 3D-consistent video generation with a maintained ecosystem, the video diffusion world models that generate in pixel space and reconstruct geometry afterwards are the comparison point, and they trade the joint RGB-and-geometry decode for a far easier install and a community around them. If you want geometry-native generation specifically, GAE's own released weights and the Depth Anything 3 backbone it freezes are the starting point regardless, and the interesting question is whether you reimplement the codec or use this.

If what you need is multi-view novel-view synthesis for evaluation rather than generation, the external estimators the eval scripts call, VGGT, MEt3R and Pi3, are the tools to reach for directly, and they need no latent design at all.

What GAE offers that those do not is the specific architectural claim: that a latent space organised for geometry produces better multi-view consistency than a pixel space with a reconstruction step bolted on. If you are evaluating that claim, this is the implementation to evaluate it against, and the paper is where the argument is made.

Editorial conclusion

Adopt GAE if you are working on camera-controllable generation or 3D-consistent video and want to start from a released implementation and checkpoint rather than build the latent design from scratch. Do not adopt it as a stable library, since it is a paper code release with a 0.1.0 version, an unpinned dataset loader path and a LICENSE file GitHub cannot classify. Before you build, confirm you have a CUDA environment on Python 3.10 to 3.12, because torch 2.5.1 pins the version range.

Frequently asked questions

Official sources

1. Issues
2. README
3. TencentARC/GAE-GeometricAutoEncoder on GitHub