# VLM3: the argument that plain vision language models already do 3D

> A Meta paper and checkpoint release that treat depth, camera pose and pixel correspondence as text output from a standard vision language model, with no new heads.

**facebookresearch/VLM3** — Official implementation of paper "VLM³: Vision Language Models Are Native 3D Learners".

- Repository: https://github.com/facebookresearch/VLM3
- Website: https://arxiv.org/pdf/2605.30561
- Stars: 438 · Forks: 14
- Language: Jupyter Notebook
- License: NOASSERTION
- Published: 2026-10-06 · Updated: 2026-10-06 · Language: en
- Canonical page: https://hysenlabs.com/projects/facebookresearch-vlm3

## The claim is that nothing needs to change but the data

The summary is unusually blunt about what it is claiming. The authors show that standard VLMs are native 3D learners, and propose VLM3 which, without complex data augmentations and any architecture or loss change, can make standard VLMs do 3D tasks.

The four task claims are specific and each names a baseline. For object-level 3D understanding, the claim is surpassing SpatialRGPT on SpatialRGPT-bench, both qualitatively and quantitatively, without using extra encoders. For metric depth estimation, matching UnidepthV2 and Moge-2, and improving the accuracy of DepthLM from 0.84 to 0.9. For pixel correspondence, surpassing DKM and RoMa. For camera pose estimation, matching DepthAnything3 and surpassing VGGT.

What the README then argues is a negative, and the negative is the interesting part. The findings say that large models, task-specific architectures, losses, data-augmentations, and even the regression formulation that sets the foundation of most state of the art 3D expert vision models are all not necessary conditions for effective 3D learning. The positive claim is that a generalist foundation model with a unified output domain of text, plus data scaling, is sufficient.

The framing that follows is a jab at the standard engineering path: you no longer need to spend a year designing complex models with different backbones, prediction heads and routings, complex losses with balancing weights across heads, or complex data augmentations like image cropping, rotation, translation and appearance augmentation. The instruction is to collect data and scale the training with a standard VLM.

## Two tricks replace the camera heads

The method overview fits in a paragraph, which is unusual for this kind of paper and is the point being made.

The first trick is focal length normalization. Given the input images, VLM3 first resizes them so the focal length is the same for all input images, with 1000 pixels given as an example. The stated reason is that this solves camera ambiguity without needing extra VLM encoders or modules. That is the load-bearing trick, because focal length ambiguity is the classic reason a monocular model needs camera parameters you were not given.

The second is that referencing an object or a pixel uses text with the pixel range normalized, for both horizontal and vertical axes, with examples such as [0, 2000) or [0, 1000). The stated advantage is that this requires no architecture and no marker rendering, and it makes the model much more flexible and scalable. So a point query becomes a piece of text like coordinates (960, 500) rather than a rendered crosshair or a special token.

The conclusion of the method section is that standard VLM architectures and text-based training, meaning supervised fine-tuning, are used to train the model. There is no new loss and no new head. The base model is named in the quickstart as Qwen3-vl-4B, and the README states that VLM3 maintains the architecture of that base model, so inference calls the model the same way you would call the original VLM.

## Inference is a chat message with coordinates in the prompt

The quickstart asks you to install transformers for inference:

```bash
pip install transformers>=5.4.0
```

The load call uses the standard image-text-to-text entry point with dtype and device_map both set to auto, and the checkpoint named in the example is `facebook/VLM3-depth`. The processor is loaded from the same name with `AutoProcessor`.

What makes this readable is the prompt. The user message carries an image and a text instruction that states the coordinates are in normalised [0, 2000] format relative to image width and height, asks for the thinking process inside `<think> </think>` tags, and asks for the final answer as the meter number only, without the unit, inside `<answer> </answer>` tags. So the answer is parsed out of a tagged span in generated text.

The generation call sets `max_new_tokens=128`, then trims the input ids off the output and batch-decodes with special tokens skipped. The whole thing is a normal chat template application followed by `model.generate`.

A detail worth noting: the coordinates are interpolated into the prompt text from `norm_x` and `norm_y` variables, so the coordinate convention is a formatting responsibility of your code rather than something the model enforces. A model that answers confidently in metres for a badly normalised query is a failure mode the prompt structure does not prevent.

## The repository is a notebook and a requirements file

The tree is short enough to list in full: `CODE_OF_CONDUCT.md`, `CONTRIBUTING.md`, `LICENSE`, `README.md`, `inference.ipynb`, `media/`, `requirements.txt` and `sample_data/`. There is no package directory, no training script, no model definition and no evaluation harness. The declared language of the repository is Jupyter Notebook.

So this is an inference release for published checkpoints, with the training story left to the paper. The README does point at a cookbook notebook at `inference.ipynb` for detailed examples on using different checkpoints for different tasks, which implies more than one checkpoint exists on HuggingFace beyond the depth one named in the quickstart.

`requirements.txt` is worth a look because of how it is written. A leading comment notes that most of these libraries can change to the newest versions and that the versions are left explicitly for full reproducibility, and then every line carries its version in a trailing comment rather than as a pin. torch 2.8.0, torchvision 0.23.0, transformers 5.4.0, tokenizers 0.22.1, accelerate 1.12.0, datasets 4.8.4, trl 1.0.0, huggingface_hub 1.5.0, safetensors 0.6.2 and sentencepiece 0.2.1 sit at the top.

Two entries there tell you about the training recipe even though the training code is absent: `liger_kernel` 0.7.0 and `flash_attn` 2.8.3, both memory-efficiency oriented for long sequences. Also present are `qwen-vl-utils` 0.0.14 for the base model's vision preprocessing, `timm` 1.0.22, and the usual data stack including numpy, pyarrow, PyYAML, Pillow and opencv-python.

The presence of trl points at supervised fine-tuning rather than reinforcement learning from human or AI feedback, which is consistent with the method description.

## Results live in images, not in numbers

The Results section of the README contains four images and no table in text: a teaser, a pipeline diagram in the method section, a visualizations figure, and two table images. The comparisons in the summary are therefore documented as figures, which means the specific numbers behind claims like matching UnidepthV2 on metric depth are not in the README text.

This is a paper-first release and the README reads like one. The header is a title with an arXiv link at 2605.30561. The abstract table at the top of the README lists a single Model column whose value is Coming Soon, which is a slightly odd artifact but at least an honest signal about the state of the release.

Attribution is a single contact, Zhipeng Cai at Meta, with a homepage and an email address given. The citation block names Cai, Zhuang Liu, Yunyang Xiong, Zechun Liu, Chandra Vikas and Yangyang Shi.

The lineage is stated too, and it helps in placing the work. The README says this work is largely motivated by the authors' previous project DepthLM, and gives a full citation for that paper, arXiv 2509.25413, titled Metric Depth from Vision Language Models. So the progression is from getting metric depth out of a VLM to generalizing the approach across the other 3D tasks.

## Licence, maturity and what to expect next

The licence is stated as FAIR CC-BY-NC, referring to the LICENSE file, which is the same non-commercial pattern used across Meta's research releases. For a project whose whole argument is that you should scale training on a standard VLM, that licence is the main constraint on who can build on it.

The repository has 430 stars, 14 forks and one open issue. That is small, and the fork count in particular is low for a release with a paper behind it, which fits a project that is weeks old rather than widely adopted. The last push was on 2026-07-26.

Seven topics are set on the repository: 3d-foundation-model, camera-pose-estimation, depth-estimation, image-matching, large-language-models, object-level-3d and vlms. Reading those as a list gives a fair summary of the scope the paper claims, and they match the four task claims plus the general framing.

What a reader should expect from the repository as it stands is inference on released checkpoints and a paper for everything else. The README does not promise training code, and the Coming Soon entry suggests the artefact set is still growing.

## Conclusion

VLM3 is a paper release with a thin repository, and that is worth saying plainly: the tree holds a README, an inference notebook, a requirements file, a media folder and sample data, with no training code, no model class and no evaluation harness. What you can actually do today is load a checkpoint from HuggingFace and prompt it, which is a real and useful thing given that the prompts are ordinary chat messages. The method's claim is that a generalist VLM with a unified text output domain plus data scale is sufficient for 3D perception, and the comparisons in the README are positioned against SpatialRGPT, UnidepthV2, DKM, RoMa, DepthAnything3 and VGGT rather than against other language models. Start with the depth checkpoint and the inference notebook, then read arXiv 2605.30561 for the training side.

## FAQ

### How do I run inference with VLM3?

Install transformers 5.4.0 or newer, load `facebook/VLM3-depth` with `AutoModelForImageTextToText` and `AutoProcessor`, build a chat message with an image and a text prompt giving normalised coordinates, then call `model.generate`. The answer is a distance in metres inside `<answer>` tags.

### Which base model does VLM3 use?

Qwen3-vl-4B. The README states that VLM3 maintains the architecture of the base model, which is why inference calls the checkpoint the same way you would call the original VLM, with no custom model class required.

### Why does the prompt ask for coordinates in a [0, 2000] range?

Because VLM3 refers to pixels as text with the range normalised for both axes, which removes the need for rendered markers or new architecture. It also resizes input images to a common focal length, given as 1000 pixels in the method overview, which is what resolves camera ambiguity without extra encoders.

### Is VLM3 training code available?

Not in this repository. The tree contains a README, an inference notebook, a requirements file, a media folder and sample data, with no training scripts or evaluation harness. The approach is documented in the paper at arXiv 2605.30561, and the requirements file lists training-adjacent libraries such as trl and liger_kernel.

## Sources

- [facebookresearch/VLM3 on GitHub](https://github.com/facebookresearch/VLM3)
- [Issues](https://github.com/facebookresearch/VLM3/issues)
- [Project website](https://arxiv.org/pdf/2605.30561)
- [README](https://github.com/facebookresearch/VLM3/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/facebookresearch-vlm3