LingBot-Depth: RGB-D Depth Completion and 3D Reconstruction

What LingBot-Depth Solves for RGB-D Systems

LingBot-Depth targets a specific weakness in RGB-D perception: a depth camera does not always return a clean, complete, geometrically consistent depth map. Missing regions, sparse measurements, sensor noise, and imperfect boundaries can make a downstream 3D reconstruction unreliable. The project states that it refines raw sensor depth into cleaner, more complete measurements and converts the result into 3D data for spatial perception.

The intended users are engineers working with robot learning, 3D vision, indoor mapping, dynamic point tracking, and dexterous manipulation. The README gives these application areas: depth completion and refinement, scene reconstruction, 4D point tracking, and dexterous manipulation. Those uses share a requirement for depth in metric space, meaning that distances have a defined physical unit rather than serving only as visual cues.

The repository provides both source code and pretrained artifacts. The model collection includes a general-purpose `v0.5` model, an earlier `v0.1` model, and a model optimized for sparse depth completion. This distinction matters because a model selected for general refinement may not be the best choice for a sparse depth sensor. The README identifies `robbyant/lingbot-depth-pretrain-vitl-14-v0.5` as the recommended general-purpose model and says that it fixes a bug in the previous version.

Masked Depth Modeling and the RGB-D Data Flow

The documented method is masked depth modeling. The model learns from incomplete depth while jointly processing RGB appearance and depth geometry in a unified latent space. This design gives the model two related signals. RGB can describe object appearance and scene structure, while the observed depth map supplies metric geometric evidence. Masking portions of the depth input creates a task in which the model must infer or repair incomplete measurements from the available context.

The architecture uses a Vision Transformer encoder with specialized depth-aware attention mechanisms. The README includes a visualization of attention from depth queries to RGB tokens, which indicates that the implementation is not simply applying an image model to a depth array. It explicitly connects depth queries with RGB features. The README excerpt does not include the exact internal equations, so the training objective, masking schedule, and decoding details cannot be confirmed from that excerpt alone.

For inference, the input flow is clear. An RGB image is converted from OpenCV's BGR layout to RGB, divided by 255, converted to a tensor, and permuted into channel-first form. Raw depth is read without changing its channels, converted to `float32`, and divided by 1000.0, which matches the repository's example convention of representing depth in meters. Intrinsics are loaded as a text matrix. `fx` and `cx` are normalized by image width, while `fy` and `cy` are normalized by image height. The model receives the RGB tensor, depth tensor, and normalized intrinsics.

The returned object includes `depth`, described as the refined depth map, and `points`, described as a 3D point cloud. The example writes these outputs as NumPy arrays and a PLY file, so the repository is oriented toward numerical inspection as well as visual review.

Installing LingBot-Depth and Running the First Example

The README documents source installation with a dedicated Conda environment using Python 3.9. The package name in `pyproject.toml` is `mdm`, and its dependencies include `torch==2.6.0`, `torchvision`, `opencv-python`, `huggingface_hub`, `numpy`, `xformers==v0.0.29.post2`, and other Python packages. A CUDA-capable GPU is recommended, although the quick-start code contains a CPU fallback for selecting a device.

Clone the repository, create the environment, and install it in editable mode:

git clone https://github.com/robbyant/lingbot-depth
cd lingbot-depth
conda create -n lingbot-depth python=3.9
conda activate lingbot-depth
python -m pip install -e .

The editable installation exposes the `mdm` package from the cloned source tree. The README says the model is downloaded from Hugging Face on first use, so there is no documented manual model-download step. The repository also includes example scenes numbered `0` through `7`.

Run the default example after installation:

python example.py

The script processes example `0` and saves its results under `result/`. The documented output includes `depth_input.npy`, `depth_refined.npy`, `depth_comparison.png`, and `point_cloud.ply`. To select another included scene, use the documented argument:

python example.py --example 1

For sparse depth completion, the README documents selecting the dedicated model:

python example.py --model robbyant/lingbot-depth-postrain-dc-vitl14-v0.5

The default output directory can be changed with `--output`, and `python example.py --help` lists the available options.

Reading the Output Files for Geometry Checks

The example's output layout is useful because it separates input, refined measurements, and visualization products. `depth_input.npy` stores the input depth as a `float32` array in meters, while `depth_refined.npy` stores the model's refined depth using the same documented unit. The `.npy` format is more suitable than a rendered image when an application needs to inspect numerical values, run quality checks, or pass the map into another geometry component.

`depth_input.png` and `depth_refined.png` provide visual comparisons, and `depth_comparison.png` places the two maps side by side. `point_cloud.ply` contains the 3D point cloud returned through the `points` output. The README does not specify the exact PLY coordinate convention, point filtering policy, treatment of invalid depth values, or file precision. Those details need to be checked against the generated file and the consuming application.

The quick-start program is also explicit about camera intrinsics. A user who already has a conventional camera matrix must normalize its focal lengths and principal point before calling `model.infer`. Omitting this step would make the camera geometry inconsistent with the preprocessing used by the example. The README does not document distortion coefficients or camera extrinsics in this quick start, so the visible interface is limited to RGB, depth, and intrinsics.

Pretrained Model Selection and Release Boundaries

The model zoo separates use cases rather than presenting one checkpoint as suitable for every input. `robbyant/lingbot-depth-pretrain-vitl-14-v0.5` is described as recommended for general-purpose depth refinement and completion across dense and sparse raw depth. `robbyant/lingbot-depth-postrain-dc-vitl14` is described as optimized for sparse depth completion, and the README also lists an earlier `v0.1` refinement model.

The project has no GitHub releases. That means there is no release history in the stated repository metadata, and users cannot rely on a documented semantic-release sequence or a repository release page for checkpoint provenance. The README's News section says the dataset became public on 2026-03-31, the paper was accepted to ECCV 2026 on 2026.06.18, and `v0.5` was uploaded on 2026.02.15. The last push was on 2026-07-08. The repository is not archived, but the absence of GitHub releases still leaves upgrade policy and rollback expectations undocumented.

The model identifiers are therefore part of the practical interface. Record the exact checkpoint identifier used by an application, including `v0.5` where applicable. The README says `v0.5` fixes a bug in the previous version, which gives a reason to prefer it over `v0.1` for new general-purpose deployments, but it does not describe the bug or provide a regression procedure.

Dataset Scale and Resource Requirements

The project publishes a curated RGB-D dataset containing 3,019,200 samples. The README breaks this into RobbyReal, with 1,400,000 real indoor samples captured using multiple RGB-D cameras; RobbyVla, with 580,960 samples collected during Vision-Language-Action robot manipulation tasks; RobbySim, with 999,264 simulated samples rendered from two camera viewpoints; and RobbySimVal, with 38,976 simulated validation samples.

This distribution exposes an important modeling choice. Training and evaluation include real and simulated data, with real robot manipulation included rather than relying only on synthetic indoor scenes. For users, that may help when their target environment resembles the captured tasks. It does not prove that the checkpoint will transfer to an arbitrary sensor, room layout, lighting condition, or camera mounting position. The README does not document sensor models, calibration procedures, train and test file schemas, or the exact split strategy beyond the dataset table.

The `pyproject.toml` pins PyTorch to `2.6.0` and xformers to `v0.0.29.post2`, while the README requirement section says Python 3.9 or newer and PyTorch 2.0 or newer. `requirements.txt` leaves most versions unspecified and includes comments referring to compatibility with other versions. This creates a concrete upgrade decision: use the editable installation described in the README first, then verify the installed CUDA, PyTorch, torchvision, and xformers versions against the environment used by the example.

When LingBot-Depth Is the Wrong Tool

LingBot-Depth is the wrong tool when the application does not have a meaningful RGB image, calibrated intrinsics, or metric depth conventions. The documented inference interface requires both RGB and depth inputs and uses camera intrinsics to interpret the depth geometry. A project that only has a monocular image or an uncalibrated depth stream would need a different preprocessing and modeling path.

It is also a poor fit for workflows that require a small, fixed, auditable release process. The repository has no GitHub releases, and the README does not document a container image, a command for exporting a model into another runtime, or a rollback procedure. The source package depends on a specific PyTorch version and an xformers build. Teams that need long-term dependency pinning should treat the editable environment as the starting point, not as a complete deployment specification.

Finally, the project is not documented as a general photogrammetry system. The available example focuses on refining an RGB-D depth map and producing a point cloud. It does not claim to solve camera calibration, visual odometry, loop closure, dynamic-scene reconstruction, or full SLAM. Those tasks require separate components or a different architecture.

Apache-2.0 and Operational Adoption Boundaries

The repository is distributed under Apache-2.0, and the README links to the `LICENSE` file. That license identifies the project's licensing basis, including the permissions and conditions associated with use, modification, and distribution. Model weights, datasets, and code should still be checked for separate notices or terms in their respective distribution locations, because the README links to Hugging Face and ModelScope artifacts without reproducing those notices here. This is a repository-level licensing observation, not legal advice.

For adoption, the first verification should be narrow and technical. Confirm that the camera produces depth in the format expected by the example, that `intrinsics.txt` follows the same matrix convention, and that the output `depth_refined.npy` is acceptable in meters for the downstream system. Then inspect `point_cloud.ply` for coordinate scale, holes, outliers, and alignment with the RGB scene. Finally, record the checkpoint identifier and dependency versions. LingBot-Depth is most defensible in a controlled RGB-D pipeline where those checks can be performed against the `v0.5` checkpoint before the model becomes part of robot or manipulation behavior.

Editorial conclusion

LingBot-Depth suits teams building RGB-D pipelines for robot learning, indoor reconstruction, dynamic tracking, or manipulation, especially when existing depth maps contain holes or noise. It is a less suitable choice when CPU-only execution, reproducible release packaging, or a clearly documented rollback path is required. Verify first that your cameras, intrinsic matrices, depth units, target scenes, and point-cloud quality meet the application's requirements.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. Robbyant/lingbot-depth on GitHub