LingBot-Vision: self-supervised ViT backbones for dense spatial perception
Self-supervised learning for spatial perception
At a glance
- What is it?
- LingBot-Vision ships four Apache-2.0 ViT backbones trained with masked boundary modeling, from ViT-S/16 to a 1.1B-parameter ViT-g/16. The package is inference-only, and the README is explicit that the released weights are backbone checkpoints, not task heads.
- Who is it for?
- Adopt LingBot-Vision if you need frozen patch-token features for depth, segmentation or video object segmentation and you are willing to write your own readout head: the release is backbone-only. Do not adopt it if you need a packaged depth or segmentation model, since LingBot-Depth 2.0 is described in the README but not released here.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 72 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem LingBot-Vision targets: features that keep object boundaries
Most self-supervised vision backbones are optimized on image-level or patch-level reconstruction, and the resulting features tend to be strong on semantics but soft on geometry. A segmentation readout built on top of them often needs a separate edge or boundary branch to recover contours. LingBot-Vision takes the opposite emphasis. According to the README, the flagship model is pretrained with masked boundary modeling, described as a boundary-centric objective that encourages spatially structured patch features while retaining strong semantic representations. The teaser caption claims the resulting features capture semantic grouping and geometric structure at the same time.
The audience is narrow and technical. The repository is a Python package of inference and PCA visualization utilities, per pyproject.toml, and the README lists dense feature visualization, depth estimation, semantic segmentation, video object segmentation and depth completion as the intended downstream uses. If your work is image classification on a single label, this is not the tool you want. If you are building a dense prediction pipeline and you want a frozen encoder whose patch tokens already carry contour information, the pitch is aimed at you.
How the backbone family and distillation are organized
The README describes a teacher-student arrangement rather than four independently trained models. A ViT-g/16 teacher with roughly 1.1B parameters is trained, and ViT-L, ViT-B and ViT-S backbones are distilled from it for inference and downstream use. That matters for evaluation: the smaller variants are not just cheaper versions of the same recipe, they inherit the teacher's boundary-centric objective through distillation.
The Giant variant uses ViT-g/16 with SwiGLU, fp32 RoPE and 4 register tokens, at an embedding dimension of 1536. Large is ViT-L/16 at 1024 dimensions and is marked in the model zoo as the recommended variant, balancing strong features against practical inference. Base is ViT-B/16 at 768, described as balanced inference cost, and Small is ViT-S/16 at 384, aimed at lightweight demos. Config files live under lingbot_vision/configs/ and, per the README, are selected automatically by load_pretrained_backbone, so you do not hand-pick a YAML for a standard variant.
Installing lingbot-vision and running a first extraction
The README states the requirements as Python 3.10 or newer, PyTorch 2.0 or newer, and a CUDA-capable GPU recommended for large-model inference. The install is a clone plus a conda environment plus two pip commands, which is the standard layout for a research package.
git clone https://github.com/robbyant/lingbot-vision.git
cd lingbot-vision
conda create -n lingbot-vision python=3.10 -y
conda activate lingbot-vision
python -m pip install -r requirements.txt
python -m pip install -e .The editable install picks up the package from pyproject.toml, whose package-data entry includes configs/*.yaml, which is why the variant configs are available after installation. Dependencies are torch, torchvision, numpy, opencv-python-headless, pillow, omegaconf and huggingface_hub, so there is no extra inference framework to learn.
The quick start loads a backbone and extracts patch tokens. The README's example uses the small variant for a lightweight smoke run, and notes that large is the default when variant is omitted.
import torch
from lingbot_vision import load_pretrained_backbone, extract_patch_tokens, load_image
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
backbone, embed_dim = load_pretrained_backbone(variant="small", device=device, dtype=dtype)
img_norm, _, _ = load_image("examples/example.png", size=512, patch_size=backbone.patch_size, mode="square")
patch_tokens, patch_grid = extract_patch_tokens(backbone, img_norm, device, dtype)
print(patch_tokens.shape, patch_grid, embed_dim)The README gives the expected output as torch.Size([1, 1024, 384]) with a (32, 32) grid and embed_dim 384. The token tensor is shaped [B, H * W, C], where H and W are the patch-grid dimensions, so a 512-pixel square input at patch size 16 yields 32 by 32 patches. The model checkpoint is downloaded from Hugging Face on first use; you can also pass a local directory or an explicit Hugging Face repo to load_pretrained_backbone. For the PCA visualization path, the README points at scripts/run_pca_demo.sh after downloading a checkpoint, though the command shown in the README is truncated.
Backbone-only weights and the missing task heads
The most consequential limitation is stated plainly in the model zoo: all released weights are backbone-only .pt checkpoints, stored as model.pt in each model repository. There is no depth head, no segmentation decoder and no video object segmentation module in this package. The README's downstream list describes what the features are good for, not what the repository ships. Depth estimation is described as frozen patch tokens feeding lightweight dense readouts, and video object segmentation as training-free token matching and label propagation with frozen features. Both of those readouts are yours to build.
A second constraint is the evaluation story. The README defers all training and evaluation details to the technical report, and the repository has no releases recorded. If you need published numbers before adopting an encoder, the paper PDF and the arXiv link are the place to look, not the repository. There is also no documented rollback or versioning path for checkpoints: the README does not describe how a checkpoint revision is pinned, so reproducibility depends on how you record the Hugging Face revision yourself.
LingBot-Depth 2.0 versus a general-purpose encoder
The natural comparison is not another self-supervised ViT but LingBot-Depth 2.0, which the README treats as a downstream consumer of this work. The difference is architectural and practical. LingBot-Depth 2.0 replaces its encoder with LingBot-Vision at the ViT-L/16 and ViT-g/16 scales and scales the curated RGB-D training corpus from 3M to 150M samples. The README claims substantial performance gains over the previous and other system, with details in the technical report.
That comparison tells you which layer you are choosing. LingBot-Depth 2.0 is a depth completion system, illustrated on mirror and glass scenes where raw sensor depth is missing and the model completes stable, contiguous surfaces across frames. LingBot-Vision is the encoder underneath, with no depth output of its own. If your problem is exactly depth completion on reflective surfaces, the encoder alone does not solve it. If your problem is a dense task the README does not cover, or you want to train your own head on frozen features, the encoder is the reusable piece and the depth system is the wrong level of abstraction. Note also that LingBot-Depth 2.0 is described in the README but is not part of this repository's top-level entries.
Licence, maintenance and upgrade cost
LingBot-Vision is Apache-2.0, declared both in the repository LICENSE file and in pyproject.toml as license = {text = "Apache-2.0"}. The README also links a LEGAL.md at the repository root, so read that file before commercial deployment; this article is not legal advice and the extra legal document is the reason to check rather than assume the licence text is the whole story.
Versioning is early. pyproject.toml declares version 0.1.0, and no releases were retrieved for this repository. The last push was on 2026-07-08, so the code has moved within the last few months, but there is no release channel to pin against. Upgrading means tracking the main branch or a specific commit, and because checkpoints are fetched from Hugging Face by name, a variant can change under you without a package version change. The practical upgrade cost is therefore not the pip install; it is re-validating your readout head whenever a checkpoint or the extraction utility changes. The dependency list is short and unpinned, which keeps installation easy but means torch and torchvision versions are resolved at install time.
Editorial conclusion
Adopt LingBot-Vision if you need frozen patch-token features for depth, segmentation or video object segmentation and you are willing to write your own readout head: the release is backbone-only. Do not adopt it if you need a packaged depth or segmentation model, since LingBot-Depth 2.0 is described in the README but not released here. Before committing, verify that the checkpoint for your chosen variant downloads from Hugging Face or ModelScope, and confirm that extract_patch_tokens returns the grid shape your downstream code expects at your input resolution.
Frequently asked questions
What is LingBot-Vision and who is it for?
It is a family of self-supervised Vision Transformer backbones for dense spatial perception, from ViT-S/16 up to a 1.1B-parameter ViT-g/16, pretrained with masked boundary modeling. It is aimed at engineers building dense downstream tasks such as depth estimation, semantic segmentation and video object segmentation on frozen patch features.
How do I install LingBot-Vision and load a pretrained backbone?
The README requires Python 3.10 or newer and PyTorch 2.0 or newer, then a clone, a conda environment, pip install -r requirements.txt and pip install -e . After that, load_pretrained_backbone downloads the checkpoint from Hugging Face on first use. Passing variant="small" is the lightweight smoke run, while large is the default.
Does LingBot-Vision include depth estimation or segmentation models?
No. The README states that all released weights are backbone-only .pt checkpoints stored as model.pt in each model repository. Depth estimation and segmentation are described as downstream uses of the frozen patch tokens, so the dense readout is something you build.
Community notes