VGGT-SLAM 2.0: dense RGB SLAM on the SL(4) manifold, and what its setup script actually pulls in

What VGGT-SLAM 2.0 is for, and who ends up using it

VGGT-SLAM solves a specific problem: turning a sequence of ordinary RGB images into a dense 3D map without a depth sensor and without a hand-built feature pipeline. The README describes it as a feedforward SLAM system optimized on the SL(4) manifold, and the repository carries two entry points, main.py for offline image folders and main_realtime.py for a live RealSense camera. The audience is therefore narrow but well defined. Researchers who already work with TUM or 7-Scenes trajectories and want to compare against MASt3R-SLAM style baselines are the primary group, because the repository ships evals/eval_tum.sh and evals/eval_7scenes.sh that automate exactly those runs. A second group is robotics people who want a dense map from a phone video as a quick sanity check on a new environment. The README's custom data section addresses them directly: record a trajectory, convert the MOV to frames, point main.py at the folder. What the project is not is a lightweight library you import into an existing pipeline. There is no published API surface in the README, and setup.py only packages evals and vggt_slam as Python packages, so the intended use is running the scripts rather than calling functions from your own code.

How the SL(4) optimization and the submap loop fit together

The mechanism visible in the repository is a submap pipeline. Images are grouped into submaps of a configurable size, the default in the evaluation scripts being 32, and the README's realtime advice suggests dropping to 8 when you want more frequent map updates. Each submap is processed by the feed-forward reconstruction stack, and the resulting submaps are then aligned. The alignment is where the SL(4) manifold comes in: rather than optimizing poses in the usual SE(3) space, the system optimizes over SL(4), which the paper title presents as the core contribution. The practical consequence for a user is that loop closure is expressed as a global optimization over submaps, and the README's sample run uses --max_loops 1 to allow a single loop closure toward the end of the office trajectory. The data flow is therefore folder of images, then submaps, then a global optimization, then a visualization in viser. Open-set object detection is a side branch: --run_os enables it, the user is prompted for text queries, and detections come back as 3D bounding boxes drawn on the map. The README is explicit that SAM 3 and Perception Encoder are used only for that optional detection path, which matters for anyone who wants to skip the extra model downloads.

Installing VGGT-SLAM 2.0 and running the office loop

The README gives a conda-based install. Clone the repository, create an environment on Python 3.11, then run the setup script, which the README says automatically downloads all third-party packages including Perception Encoder, SAM 3, and the project's fork of VGGT.

git clone https://github.com/MIT-SPARK/VGGT-SLAM
cd VGGT-SLAM
conda create -n vggt-slam python=3.11
conda activate vggt-slam
chmod +x setup.sh
./setup.sh

That last step is the one to plan around. It is not a pip install from requirements.txt; it reaches out to three external model repositories, and the README links a separate licence for each of Perception Encoder, SAM 3, and VGGT. If you only want the reconstruction path and not object detection, the README still describes setup.sh as downloading all of them, so the download happens either way.

The quickest real run uses the sample data that ships in the repository root. Unzip office_loop.zip, then point main.py at the resulting folder with a single allowed loop closure and the viser map enabled.

unzip office_loop.zip
python3 main.py --image_folder office_loop --max_loops 1 --vis_map

The README states this produces a visualization in viser showing the incremental construction of the map, and that the default parameters give a single loop closure toward the end of the trajectory. If you want to try the object queries, add --run_os and type one of the README's example queries such as "coffee machine" or "printer" when prompted.

For your own footage, the README's recipe is to record with a phone and convert to frames at 10 fps. It notes that horizontal video is recommended to avoid cropping.

mkdir <desired_location>/img_folder
ffmpeg -i /path/to/video.MOV -vf "fps=10" <desired_location>/img_folder/frame_%04d.jpg

For live capture, the README says to plug in a RealSense camera and run main_realtime.py with --vis_map, adding --submap_size 8 if you want more frequent map updates.

Where the visualizer and the submap size become the real constraints

The most concrete limitation in the README concerns the visualizer, not the SLAM backend. Displaying all 3D points in viser can slow it down or crash it on larger datasets. The suggested mitigation is --vis_voxel_size 0.005, and the README is careful to note that increasing that number decreases the number of displayed points while not affecting the number of points stored or used internally. That distinction is worth reading twice, because it means the visualizer is a diagnostic window and not a representation of what the system actually holds. A second visualizer cost is images: by default only camera frustums are drawn, and --vis_imgs adds the images at the price of a slower visualizer. Neither of these is a bug, but together they mean the thing you look at during a run is a deliberately thinned view of the map.

The other constraint is submap size. The README treats it as a tuning knob in two directions: larger values in the evaluation scripts, smaller values for realtime runs. There is no guidance in the README on how submap size interacts with loop closure quality or with GPU memory, and no stated minimum or maximum. That is a genuine gap. Anyone running this on long trajectories will be choosing that number by experiment rather than from documentation.

The case where VGGT-SLAM is the wrong tool is monocular scale. The README's custom data path is a single phone video converted to frames, and nothing in the README describes metric scale recovery or an IMU input. If your application needs distances in metres from a monocular capture, this pipeline as documented does not give you that. Likewise, if you need a stable Python API to embed in a larger system, the repository layout does not suggest one.

How it differs from MASt3R-SLAM, the baseline it borrows from

The README points at MASt3R-SLAM for dataset download instructions, which makes the comparison natural. MASt3R-SLAM is a dense SLAM system built on the MASt3R matching and reconstruction model, and the VGGT-SLAM evaluation scripts reuse its dataset layout by asking you to set abs_dir in evals/eval_tum.sh and evals/eval_7scenes.sh to the MASt3R-SLAM download location. The difference in approach is the optimization domain. MASt3R-SLAM works with pose and pointmap estimation in the usual rigid-body formulation, while VGGT-SLAM 2.0 optimizes on the SL(4) manifold, which is what its title advertises and what the news section says was integrated into the official GTSAM repository in August 2025. There is also a packaging difference: VGGT-SLAM depends on gtsam-develop in requirements.txt, so the optimizer comes from the GTSAM ecosystem rather than being self-contained. If you already have a MASt3R-SLAM dataset tree on disk, the VGGT-SLAM evaluation scripts are designed to reuse it, which makes a side-by-side comparison on TUM or 7-Scenes the cheapest way to judge whether the manifold formulation buys you anything on your scenes.

Maintenance, licence and what upgrading costs you

The repository is not archived and the last push was on 2026-06-29, so the code has moved recently. That said, the project publishes no releases, and the README's news section is the only changelog. It records VGGT-SLAM 1.0 in May 2025, the SL(4) integration into GTSAM in August 2025, acceptance to NeurIPS 2025 in September 2025, VGGT-SLAM 2.0 in January 2026, acceptance to RSS 2026 in May 2026, a follow-on project called FOUND-IT in May 2026, and realtime code plus visualizer upgrades in June 2026. The upgrade cost follows from that structure: version 1.0 lives on the version1.0 branch of the same repository, so switching between the two generations is a branch checkout rather than a package version bump. There is no migration guide in the README, and no documented rollback procedure.

On licensing, the repository itself is BSD-2-Clause. That is only part of the picture. The README states that setup.sh downloads Perception Encoder, SAM 3 and a fork of VGGT, and links a separate licence for each. Those are third-party licences with their own terms, and the repository's BSD-2-Clause grant does not override them. Whether they are compatible with your intended use is a question for your own review; the README does not attempt to answer it. The requirements.txt also pins torch==2.3.1 and torchvision==0.18.1 alongside viser==0.2.23, so a CUDA or Python upgrade in your environment can force a dependency resolution you did not plan for.

Editorial conclusion

Adopt VGGT-SLAM 2.0 if you have a CUDA machine, a folder of RGB frames or a RealSense camera, and you want a dense map plus an optional open-set object query on top of it; the office_loop.zip sample lets you check the whole pipeline before spending time on your own capture. Do not adopt it if you need metric scale from monocular video, if you cannot accept the third-party model licences that setup.sh downloads, or if you need a documented upgrade path between releases, because the repository lists no releases at all. Verify three things before committing: that your GPU memory fits the submap size you intend to use, that the SAM 3 and Perception Encoder licences are acceptable for your deployment, and that the ffmpeg frame rate you choose gives you enough frames per submap for the loop closure you are aiming for.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-2-Clause
3. MIT-SPARK/VGGT-SLAM on GitHub
4. README