# X-Lens: one depth model for pinhole, fisheye and mixed camera rigs

> X-Lens is a metric depth estimation model that consumes perspective, wide field of view fisheye and heterogeneous camera views in a single forward pass, fusing them into one point cloud, and it ships inference code with checkpoints for all three camera modes. It suits a robotics or driver perception team with a mixed camera rig, and the weights are published under a non-commercial licence.

**D-Robotics-AI-Lab/XLens** — X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras

- Repository: https://github.com/D-Robotics-AI-Lab/XLens
- Stars: 565 · Forks: 8
- Language: Python
- License: NOASSERTION
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/d-robotics-ai-lab-xlens

## One model, and the camera type comes from the manifest

X-Lens recovers dense metric depth and a fused 3D point cloud from an arbitrary set of views captured by an arbitrary mix of cameras, and the claim that matters is the second half. Perspective pinhole, wide field of view fisheye, or a heterogeneous rig that combines both in a single pass, all handled by one model in one forward. The mechanism is described as a per-pixel ray representation with camera type conditioning, so there is no rectification step and no per-camera retraining. The camera type is injected per view through calibration tokens plus a geometric distortion bias. The output is not just a depth map. A learned scale head produces metric depth, and a confidence map comes with it, and every view unprojects into a single consistent point cloud that is view-0 canonical. The three supported configurations are pure pinhole with six cameras using 3x3 intrinsics, pure fisheye with four cameras using per-pixel unit rays, and heterogeneous with six cameras mixing both kinds of intrinsics. All three run from the same checkpoint, and the mode is chosen purely by what each view carries in the manifest.

## The install path is a conda environment and an editable pip install

The quick start is short and the environment is pinned to a specific Python version through conda.

```bash
git clone https://github.com/zhouhengamerica/XLens.git xlens
cd xlens
conda create -n xlens python=3.10 -y
conda activate xlens
pip install -r requirements.txt
pip install -e .          # exposes the `xlens` package
```

Note that the clone URL in the quick start points at a different GitHub account than the repository this readme is published in, which is the kind of detail worth checking before you script a reproducible setup. The dependency list is short and is the standard computer vision set: torch at 2.1 or newer, torchvision, numpy, pillow, pyyaml, einops and safetensors, with matplotlib and imageio marked optional for previews and fisheye EXR look-up tables, and onnx and onnxruntime marked optional for export and inference, with a note that onnxruntime-gpu substitutes for CUDA or TensorRT execution. The package metadata sets the project name to xlens at version 0.1.0 and requires Python 3.9 or newer, and it declares the same dependency list plus optional groups named viz and onnx.

## Weights come from Hugging Face, with the backbone bundled in

The checkpoints are downloaded with the Hugging Face command line tool into a checkpoints directory.

```bash
mkdir -p checkpoints
hf download henryzhou998/X-Lens --local-dir checkpoints
```

The detail that saves a step is called out separately. The DINOv2 backbone weights are bundled inside the checkpoint, so no separate download is needed for inference. There is also a model zoo referenced from the quick start for the checkpoint list, and a Hugging Face dataset release, the OmniScene dataset, published on the same day as the initial release. The news entries all carry the date 07-15-2026, and they cover three things: the OmniScene dataset release, the initial release with inference code and XLens-S checkpoints for all three camera modes, and the paper on arXiv. That combination of same-day dataset, code, weights and paper is unusual and worth noting, because it means you can reproduce the reported setting rather than approximating it.

## A JSON manifest per view, and what comes out

A scene is a small JSON manifest listing, per view, an image and its calibration, plus an optional pose. Each view carries an image, a c2w pose, an optional mask, and one of three calibration forms: K for 3x3 pinhole intrinsics, dcam which is a .npy file of per-pixel unit rays for any camera model, or lut which is a fisheye ray look-up table. The cam_type value, zero for fisheye and one for pinhole, is inferred from the calibration or set explicitly, and the CLI assigns the camera type automatically. The run itself is one command with a checkpoint, a manifest, an output directory and two thresholds.

```bash
python inference.py \
    --ckpt checkpoints/xlens.pth \
    --manifest examples/hetero_loft/scene.json \
    --out out/hetero --max-depth 25 --conf-drop-pct 10
# -> out/hetero/depth_metric.npy   (6, 504, 798) metric depth
# -> out/hetero/depth_preview.png  per-view depth mosaic
# -> out/hetero/points.ply         fused world-frame point cloud
```

The three outputs are the metric depth array, a per-view depth mosaic for inspection, and the fused point cloud in PLY format. The example shown is the heterogeneous scene with four fisheye and two pinhole cameras, and the depth array shape of 6, 504, 798 corresponds to the six views. The manifest format is not a detail you can ignore, because it is what tells the model which camera model each view is.

## Fisheye edges need cleaning, and the tool does it for you

The readme is unusually explicit about a failure mode, and that is the section a robotics engineer should read first. Near a fisheye lens's edge, distortion is extreme and per-pixel depth becomes unreliable, and those peripheral rays unproject into stray points that shoot outside the scene. When a scene contains any fisheye view, inference.py applies the cleanup automatically. It trims rays more than 85 degrees off the optical axis and drops the lowest 8 percent of points by confidence, and the same two thresholds can be set on the command line with the fov-max and conf-drop-pct options. The plain pinhole command does not need them. So the fisheye examples carry the fov-max 85 and conf-drop-pct 8 flags while the pinhole example only sets a max depth of 15. The bundled demo scenes are self-contained, with RGB images plus intrinsics and extrinsics plus masks, and they are named for their source: a pinhole capture from OmniOCC, an FS3 FuelBar fisheye set, and the heterogeneous loft scene with four fisheye and two pinhole cameras.

## Licence and release state are the two things to check before you build

The licence badge on the readme names CC BY-NC 4.0, which is a non-commercial licence, and the repository listing records the licence as a custom one that GitHub cannot classify. For a research group reproducing a paper result, that is workable. For a company putting a depth model on a robot or in a vehicle, it is a blocker until you reach whoever holds the rights, and the file to read is the LICENSE file at the top of the repository. The release state is the second thing. The repository is not archived and the last push was on 2026-07-29, but there are no GitHub releases attached, and the news section shows a single initial release dated 07-15-2026, so you should treat the current tree as the artifact rather than a tagged version. The project comes from D-Robotics with affiliations at the University of Tokyo and Soochow University, and the tree holds the expected parts for a research release: an inference.py entry point, an xlens package directory, configs, an evaluation directory, the three example scenes, and a requirements file with a pyproject beside it.

## Conclusion

X-Lens fits a team whose rig mixes wide field of view fisheye cameras with pinhole cameras and who want metric depth and a fused point cloud from a single model rather than a rectification step and a per-camera retrain, which is the specific gap the project targets. It does not fit a commercial deployment, because the published licence is CC BY-NC 4.0, and it does not fit a team that needs a calibrated production pipeline, since the release is inference code with checkpoints and a paper rather than a training recipe. Before you build on it, check the licence against your use case, run the bundled demo scenes to see the output on your own camera geometry, and decide whether the automatic fisheye cleanup thresholds suit your rig, because the defaults trim rays beyond 85 degrees and drop the least confident 8 percent of points.

## FAQ

### What is X-Lens and what problem does it solve?

X-Lens recovers dense metric depth and a fused 3D point cloud from an arbitrary set of views captured by an arbitrary mix of cameras, perspective pinhole, wide field of view fisheye, or a heterogeneous rig combining both. One model and one forward pass handle all of them, with no rectification and no per-camera retraining.

### How do I install X-Lens and get the model weights?

Clone the repository, create a conda environment with python=3.10, activate it, then run pip install -r requirements.txt and pip install -e . The checkpoints are fetched with hf download henryzhou998/X-Lens --local-dir checkpoints, and the DINOv2 backbone weights are already bundled inside the checkpoint.

### What is in an X-Lens scene manifest?

Each view carries an image, a c2w pose, an optional mask, and one of three calibration forms: K for 3x3 pinhole intrinsics, dcam as a .npy of per-pixel unit rays, or lut for a fisheye ray look-up table. The cam_type value, zero for fisheye and one for pinhole, is inferred from the calibration or set explicitly.

### Why does X-Lens clean up fisheye depth automatically?

Because near a fisheye lens's edge distortion is extreme and per-pixel depth becomes unreliable, so those peripheral rays unproject into stray points outside the scene. When any fisheye view is present, inference.py trims rays more than 85 degrees off the optical axis and drops the lowest 8 percent of points by confidence.

### What outputs does X-Lens produce?

A metric depth array saved as depth_metric.npy, a per-view depth mosaic saved as depth_preview.png, and the fused world-frame point cloud saved as points.ply, all written into the directory given by the out option.

## Sources

- [D-Robotics-AI-Lab/XLens on GitHub](https://github.com/D-Robotics-AI-Lab/XLens)
- [Issues](https://github.com/D-Robotics-AI-Lab/XLens/issues)
- [README](https://github.com/D-Robotics-AI-Lab/XLens/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/d-robotics-ai-lab-xlens
