VGGT-Omega: gated checkpoints, a 1B feed-forward reconstruction model, and what the README leaves out

What VGGT-Omega predicts, and who is meant to run it

VGGT-Omega takes a set of images and returns, in one forward pass, camera extrinsics and intrinsics, a depth map per image, a depth confidence map, and a set of camera and register tokens. The README describes the package as "VGGT-Omega feed-forward camera and depth reconstruction", and the quick-start snippet shows the whole pipeline: load images, run the model, convert `pose_enc` into a camera with `encoding_to_camera`. There is no bundle adjustment, no feature matching stage and no iterative solver in the documented path.

The intended audience is fairly narrow. The authors are at the Visual Geometry Group at Oxford and Meta AI, the repository carries a CVPR 2026 Oral tag, and the training code, dataset preparation guides and a curation pipeline all ship in the same tree. That points at researchers and engineers who want to reproduce or fine-tune a reconstruction model, not at someone who wants a drop-in photogrammetry service. A separate Gradio demo exists for people who only want to see output, and the README notes the hosted Hugging Face demo "is available to everyone" while the checkpoints are not.

How the pipeline is wired: images in, pose encoding out

The data flow is short and worth stating precisely, because the naming is where most confusion starts. `load_and_preprocess_images` reads the image paths and returns a tensor at the requested resolution. The model returns a dictionary. Three keys matter most: `pose_enc`, `depth` and `depth_conf`. Camera parameters do not come back as matrices directly; `encoding_to_camera` converts `pose_enc` into extrinsics and intrinsics, and it needs the spatial shape of `predictions["images"]` to do that conversion.

The model also exposes `camera_and_register_tokens`, which the snippet splits into `camera_tokens` (the first slot) and `registers` (everything after). Those are internal representations, not reconstruction output, and the README does not explain what to do with them beyond handing them back.

The text-aligned checkpoint changes the call signature rather than the pipeline: construct with `VGGTOmega(enable_alignment=True)`, use `image_resolution=256`, and read `predictions["text_alignment_embedding"]`. Only one of the three released checkpoints supports this, so alignment is a per-checkpoint capability, not a model-wide feature.

Installing VGGT-Omega and running a first inference

The README gives a clone-and-install sequence. It uses the SSH clone URL, so you need a GitHub key configured, or you swap in the HTTPS URL yourself.

git clone git@github.com:facebookresearch/vggt-omega.git
cd vggt-omega
pip install -r requirements.txt
pip install -e .

`requirements.txt` pins `torch>=2.6`, `torchvision>=0.21`, `numpy<2`, `Pillow`, `einops`, `safetensors` and `opencv-python`. The `numpy<2` cap is the one to watch if you are installing into an environment that already has NumPy 2.x. `pyproject.toml` declares `requires-python = ">=3.10"`. Note that torch is not in the base `dependencies` list in `pyproject.toml`; it arrives through `requirements.txt`.

The README's own example is the shortest path to a real result:

import torch

from vggt_omega.models import VGGTOmega
from vggt_omega.utils.load_fn import load_and_preprocess_images
from vggt_omega.utils.pose_enc import encoding_to_camera

checkpoint_path = "path/to/vggt_omega_1b_512.pt"
image_names = ["path/to/imageA.png", "path/to/imageB.png", "path/to/imageC.png"]

model = VGGTOmega().to("cuda").eval()
model.load_state_dict(torch.load(checkpoint_path, map_location="cpu"))

Before any of that runs, you need the weights, and this is the first real obstacle. The README instructs you to request access to the checkpoints on Hugging Face and states that requests are reviewed by an automated process, that the authors are not involved, and that they cannot approve or reject individual applications. There is no documented manual fallback if the automated review declines.

For a visual check without writing code, the demo has its own dependency file and a launch script:

pip install -r requirements_demo.txt

python demo_gradio.py \
  --checkpoint checkpoints/VGGT-Omega-1B-512/model.pt \
  --image-resolution 512

The demo accepts uploaded images or a video and renders the depth-unprojected point cloud and predicted cameras as a GLB scene. The flags shown are the ones the README uses; nothing else is documented.

GPU memory scales with frame count, and the README stops at 500

The memory table is the most useful thing in the repository, and also the most limiting. It reports peak GPU memory for `VGGT-Omega-1B-512` on a single NVIDIA A100 with 624x416 inputs, measured end to end from loading weights through the forward pass. One frame costs 6.02 GB. Ten frames cost 6.67 GB. Fifty frames cost 9.66 GB. One hundred frames cost 13.37 GB, two hundred cost 20.82 GB, and five hundred cost 43.15 GB.

The shape of that curve matters more than any single number. Between one and ten frames the cost barely moves, because the model weights dominate. Past that, memory grows close to linearly with frame count, so a 24 GB card sits somewhere between 200 and 300 frames under this configuration. The table stops at 500 frames and the README does not say what happens beyond it.

There is a documented way to trade resolution for memory: `mode="max_size"` resizes the longest side to 512 instead of using the default `mode="balanced"`. The README states that for the same roughly 3:2 aspect ratio this gives about 512x336 inputs rather than 624x416. That is a real reduction in pixel count, and the README does not report a memory table for it, so you would be measuring that yourself.

One caveat about the benchmark: it is an A100 measurement, and the README does not state a minimum supported GPU, a CPU path, or behaviour under `torch.cuda.OutOfMemoryError`. If your card has less available memory than the frame count requires, the documented path gives you no guidance.

Three checkpoints, two of them not for benchmarking

The checkpoint table is easy to misread. `VGGT-Omega-1B-512` runs at 512 resolution with no text alignment and is marked "Recommended for in-the-wild applications, no benchmarking". `VGGT-Omega-1B-256-Text-Alignment` runs at 256, supports text alignment, and carries no note. `VGGT-Omega-1B-416-Reproduction` runs at 416 and is the one the release notes say "should serve as the reference for future comparisons on the reported benchmarks".

That last point is a correction, and the README says so plainly. The September 8 update states the reproduction checkpoint was released "to further validate reproducibility and address a potential concern with the original checkpoint". If you are comparing against published numbers, using the 512 checkpoint would be comparing against the wrong artifact. The reproduction details live in `reproduction.md`, which the README links but does not summarise.

The 512 checkpoint is the one the memory table uses and the one the demo example names. So the artifact with the best documented resource profile is explicitly not the one for benchmark comparison. That split is the single most consequential thing to get right before you start.

Where VGGT-Omega is the wrong tool

The clearest limitation is licensing, and it is unresolved. The repository's LICENSE file is present, but the metadata reports the licence as NOASSERTION, meaning no standard identifier was detected. The README does not restate the terms. If you need a model you can redistribute inside a product, this repository does not tell you whether you can, and the gated checkpoint download adds a second layer of terms on top of the code licence. Treat licence review as a prerequisite, not a follow-up.

Second, access is gated. The code installs with pip in a few commands, but nothing runs without an approved Hugging Face request, and the README states the authors cannot intervene in that process. Teams that need a reproducible environment built today, without waiting on an external approval queue, are not served well by this setup. The hosted demo is the only documented path that avoids the gate, and it is a demo, not an API.

Third, the documentation is silent on failure. There is no section on what happens with few images, images with no overlap, reflective or textureless surfaces, or degenerate camera configurations. There is no stated accuracy metric in the README, only a memory table. For a reconstruction model, the absence of a documented error mode is a bigger gap than the absence of a benchmark, because you will discover the failure boundaries in your own data rather than in the docs.

Finally, the hardware floor is real. With a 6.02 GB baseline for a single frame, this is not a model you run casually on a laptop GPU, and no CPU path is documented.

What to compare it against: VGGT and classical SfM

The obvious comparison is the earlier VGGT line, and the naming invites it. The difference visible in this repository is scope rather than architecture: VGGT-Omega ships training code, dataset preparation guides covering collection, conversion and cleaning, a supervised geometric filtering pipeline under `training/curation`, sequence lists for eight datasets under `training/valid_seqs`, and a reannotated UCo3D dataset being uploaded to Hugging Face. That is a reproduction and training package wrapped around the inference model. If you only want inference, most of that tree is irrelevant to you.

The other comparison is classical structure-from-motion. Tools in that family typically match features across images, estimate poses incrementally, and refine everything with bundle adjustment. VGGT-Omega does none of that in the documented path: one forward pass produces poses, depth and confidence. The trade is control for speed. Classical pipelines expose the intermediate state, so you can inspect a bad match or a failed bundle adjustment. VGGT-Omega returns tensors and a confidence map, and the README does not describe how to diagnose a bad result beyond looking at `depth_conf`.

There is a bridge between the two worlds: `pyproject.toml` defines an `export` extra that pulls in `pycolmap>=3.10.0`, which suggests the intended route for taking predictions into a COLMAP-compatible workflow. The README does not document that export path, so the extra is a hint rather than a tutorial.

Maintenance, training cost and licence exposure

The repository is not archived, and the last push was on 2026-09-09. The update log shows a dense run of activity in early September 2026: training code and a reproduction checkpoint on September 8, dataset preparation guides and the curation pipeline on September 9, and the eight-dataset sequence lists on September 10. That is a project in the middle of an active release cycle, not a frozen snapshot. It also means the surface is moving. `pyproject.toml` still declares `version = "0.0.1"`, and no releases were retrieved, so there is no tagged version to pin against. If you build on this, pin a commit hash rather than tracking `main`.

Upgrade cost is dominated by the training extras rather than the inference path. The `train` extra pulls `torch>=2.6`, `hydra-core>=1.3`, `omegaconf>=2.3`, `fvcore`, `tensorboard` and `wcmatch`. The separate `curation` extra pulls `catboost`, `joblib`, `lightgbm`, `scikit-learn>=1.4`, `scipy` and `xgboost`, which is a gradient-boosting stack sitting alongside a deep-learning stack. Both are optional, and neither is needed for inference. The `numpy<2` pin in the base requirements is the constraint most likely to collide with an existing environment.

On licensing, the honest position is that no conclusion is supported. The LICENSE file exists at the repository root, `pyproject.toml` points at it with `license = {file = "LICENSE"}`, and the metadata reports NOASSERTION. The README does not summarise the terms, and the checkpoints carry their own gated access conditions on Hugging Face. Read the LICENSE file and the checkpoint terms directly before shipping anything, and do not treat the presence of a LICENSE file as permission.

Editorial conclusion

Adopt VGGT-Omega if you need multi-image camera and depth output from a single forward pass and you can wait for a Hugging Face access approval, or if the hosted demo covers your evaluation. Do not adopt it if you need CPU-only inference, a permissively licensed redistributable model, or a documented failure-mode section; the README provides none of those. Verify first that your GPU has enough free memory for your frame count, that the checkpoint you request is the one you intend to benchmark against, and that the LICENSE file actually grants the rights your product needs.

Frequently asked questions

Official sources

1. facebookresearch/vggt-omega on GitHub
2. Issues
3. README