# Fast-FoundationStereo for real-time zero-shot stereo matching

> Fast-FoundationStereo is NVIDIA's official implementation for real-time zero-shot stereo matching, aimed at engineers who need foundation-model generalization without offline-scale inference cost. The repository offers several checkpoints and tuning controls, while its custom licensing and calibration requirements need review.

**NVlabs/Fast-FoundationStereo** — [CVPR 2026] Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching

- Repository: https://github.com/NVlabs/Fast-FoundationStereo
- Website: https://nvlabs.github.io/Fast-FoundationStereo/
- Stars: 1,587 · Forks: 168
- Language: Python
- License: NOASSERTION
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/nvlabs-fast-foundationstereo

## Why Fast-FoundationStereo exists

Fast-FoundationStereo addresses a specific tension between generalization and latency in stereo matching. According to the README, stereo foundation models provide strong zero-shot generalization but remain too expensive for real-time applications. The authors characterize efficient stereo architectures as faster, but say they sacrifice robustness and require costly per-domain fine-tuning. The project is intended for engineers who need a stereo model to work across domains without training a separate model for each deployment environment.

The official Python repository implements a family of architectures described in a CVPR 2026 paper. It is not presented as a general-purpose image depth API. Its inputs are a rectified left image, its matching right image, and camera calibration information. The demo can estimate disparity and produce a point cloud, so the likely users are developers working on robotic perception, real-time reconstruction, depth systems, or stereo sensor pipelines.

The README explicitly warns against using this model when offline computation and the highest possible accuracy are the priorities. In that case, it points to NVlabs' earlier FoundationStereo project instead. That distinction matters. Fast-FoundationStereo is designed around a latency budget, and its checkpoints expose different compromises between runtime and accuracy. It should not be treated as an unrestricted replacement for FoundationStereo in batch processing.

The repository is not archived, and its last push was on 2026-09-11. There are no GitHub releases, so the repository does not provide a conventional release history through GitHub. Maintenance and upgrade planning must account for that packaging state.

## Distillation, architecture search, pruning, and pseudo-labels

The documented acceleration strategy has three model-related components. First, knowledge distillation compresses a hybrid backbone into a single efficient student. The purpose is to retain zero-shot behavior while reducing inference work. Second, blockwise neural architecture search searches for cost-filtering designs under latency constraints. The README says blockwise search reduces search complexity exponentially, although it does not document the search implementation or provide a separate search command.

Third, structured pruning removes redundancy from the iterative refinement module. Refinement is user-controlled through `valid_iters`, which the demo describes as the number of refinement updates performed during the forward pass. This control gives operators a direct way to trade runtime for output quality. In the published checkpoint table, changing `valid_iters` from 8 to 4 lowers the reported runtime but does not materially change the listed peak memory. Those numbers apply only to the documented image size and RTX 3090 profiling context.

Training uses a separate data pipeline. The authors describe an automatic pseudo-labeling system that curates 1.4 million in-the-wild stereo pairs to supplement synthetic training data and support knowledge distillation. The README does not document how those pairs are collected, filtered, licensed, or reproduced by users. It also does not expose a training command in the documented setup instructions. As a result, the repository clearly supports inference, but the documentation does not establish that the full data-curation or training workflow can be recreated from it alone.

The final architecture and learned weights are still required to reproduce the paper's claimed behavior. Source code alone does not eliminate the checkpoint and environment requirements described below.

## Docker and pip setup for Fast-FoundationStereo

The README offers two documented installation routes. Docker builds an image named `ffs` from the repository's `docker/dockerfile`, then starts it through the included shell script. These commands use the repository root as the build context.

```bash
docker build --network host -t ffs -f docker/dockerfile .
```

The image should build from the checked-out repository. The next documented command launches the container through the project's wrapper script.

```bash
bash docker/run_container.sh
```

The pip route requires Python 3.12 and specific PyTorch packages built for CUDA 12.4.

```bash
conda create -n ffs python=3.12 && conda activate ffs
pip install torch==2.6.0 torchvision==0.21.0 xformers --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
```

`requirements.txt` adds `timm`, `einops`, `omegaconf`, `scipy`, `numpy`, `scikit-image`, `opencv-contrib-python`, `imageio`, `pyyaml`, and `open3d`. It also lists TensorRT-related packages as commented optional entries. The README does not provide commands for enabling those entries, so an engineer should not assume TensorRT is configured by the base pip installation.

The runtime table reports separate PyTorch and TensorRT figures, but the README does not explain the complete export workflow. The basic demo uses Python inference.

## Selecting weights and running the first stereo demo

Research checkpoints are distributed through Google Drive and should be placed under `weights/`, for example at `./weights/23-36-37`. The README separates research checkpoints from the commercially licensed C-Fast-FoundationStereo model on Hugging Face. The required file is also named explicitly in the demo command.

The following command uses the `23-36-37` checkpoint with the included demonstration images. It enables point-cloud generation and cloud denoising, uses eight refinement iterations, and writes results to `output/`.

```bash
python scripts/run_demo.py --model_dir weights/23-36-37/model_best_bp2_serialize.pth --left_file demo_data/left.png --right_file demo_data/right.png --intrinsic_file demo_data/K.txt --out_dir output/ --remove_invisible 0 --denoise_cloud 1 --scale 1 --get_pc 1 --valid_iters 8 --max_disp 192 --zfar 100
```

The command writes results under `output/`, although the README does not enumerate every generated file.

Several parameters affect the result. `--remove_invisible 0` retains depth for the non-overlapping region, while changing it to `1` excludes that region. `--get_pc 1` requests a point cloud, and `--denoise_cloud 1` applies denoising. `--max_disp 192` sets the maximum disparity used for volume encoding. The README says 192 should be sufficient unless very near objects must be sensed, such as objects below 0.1 meters. Raising it makes inference slower and uses more memory. `--zfar` controls the maximum depth included in the point cloud.

For a first run, keep the documented values and inspect the generated outputs before changing speed or geometry settings.

## Rectification, calibration, and operational failure modes

The most immediate failure mode is supplying images that do not satisfy the stereo geometry expected by the model. The README says the left and right images must be rectified and undistorted. Fisheye distortion should be removed, and epipolar lines should be horizontal. Images from cameras such as the Zed are said to already perform this processing in typical cases, but the repository does not include a general camera calibration tutorial.

Image order is equally important. The left file must come from the left-side camera. The README warns against swapping the two images and notes that objects in the actual left image should appear on the right side. Reversing the pair can therefore produce incorrect correspondence rather than a clear input error.

The project recommends PNG files without lossy compression. It says stereo RGB is the preferred input, while monochrome and infrared pairs from devices such as RealSense D4XX have also been tested by the authors. That statement supports trying those sensor types, but it does not specify supported resolutions, camera models, exposure conditions, or calibration tolerances.

`--max_disp` introduces another concrete limit. The default example uses 192. Objects closer than roughly 0.1 meters may require a higher value, while doing so increases runtime and memory consumption. `--zfar` separately removes geometry beyond a selected maximum depth from the point cloud. These settings can produce plausible files while omitting relevant geometry, so output completeness should be checked against the expected operating range.

The README does not document rollback behavior, service deployment, or automated recovery after interrupted inference.

## FoundationStereo, licensing, and maintenance costs

The clearest alternative is NVlabs' FoundationStereo, which the README explicitly recommends for offline computation and best accuracy. Fast-FoundationStereo changes the approach by distilling a hybrid foundation-model backbone into an efficient student, searching cost-filtering blocks under latency limits, pruning iterative refinement, and training with additional pseudo-labeled stereo pairs. FoundationStereo is therefore the accuracy-oriented choice, while Fast-FoundationStereo is organized around lower inference cost.

License review is a separate adoption issue. The repository is marked with a custom license that GitHub cannot classify, and the README links to LICENSE.txt. Research checkpoints are linked through Google Drive. Commercial model use points to C-Fast-FoundationStereo under the NVIDIA Open Model Agreement, while commercial training and inference code are directed to the NVIDIA TAO Toolkit. The README does not reproduce those terms, so it supports neither a legal conclusion nor an assumption that all checkpoint and code use is interchangeable.

Maintenance is recent, with the last push on 2026-09-11, but GitHub lists no releases. Upgrade work may therefore involve repository changes and checkpoint compatibility rather than selecting a named GitHub release. The pinned Python, PyTorch, torchvision, CUDA wheel index, and model files create an explicit environment matrix to preserve. `valid_iters`, checkpoint selection, TensorRT use, camera calibration, and commercial terms should each be versioned in an internal deployment record.

## Conclusion

Fast-FoundationStereo fits teams building rectified stereo pipelines that need lower-latency zero-shot matching and can provide suitable checkpoints and GPU software. It is a weaker choice for offline work focused on maximum accuracy, for which the README directs users to FoundationStereo, or for organizations that require plainly permissive licensing without first reviewing LICENSE.txt and the NVIDIA Open Model Agreement. Verify the weight license, CUDA and xformers compatibility, calibration format, rectification, and acceptable accuracy after changing `valid_iters` before integrating the model into production.

## FAQ

### How do I install Fast-FoundationStereo?

The README provides Docker and pip routes. The pip route creates a Python 3.12 Conda environment, installs the documented PyTorch packages, and then installs `requirements.txt`.

### How do I run the Fast-FoundationStereo stereo demo?

Place a research checkpoint under `weights/`, then run `python scripts/run_demo.py` with the checkpoint, left and right images, intrinsic file, output directory, and geometry settings. The repository includes `demo_data/left.png`, `demo_data/right.png`, and `demo_data/K.txt` for the documented example.

### Can I use Fast-FoundationStereo for commercial applications?

The README directs commercial model use to C-Fast-FoundationStereo under the NVIDIA Open Model Agreement. It separately directs commercial training and inference code to the NVIDIA TAO Toolkit, so users must review both paths before use.

### What input does Fast-FoundationStereo need?

It needs rectified and undistorted left and right images plus an intrinsic matrix and baseline file for point-cloud output. The README recommends lossless PNG input and warns not to swap the camera order.

## Sources

- [Issues](https://github.com/NVlabs/Fast-FoundationStereo/issues)
- [NVlabs/Fast-FoundationStereo on GitHub](https://github.com/NVlabs/Fast-FoundationStereo)
- [Project website](https://nvlabs.github.io/Fast-FoundationStereo/)
- [README](https://github.com/NVlabs/Fast-FoundationStereo/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nvlabs-fast-foundationstereo