# INSID3: in-context segmentation on a frozen DINOv3, with no training

> INSID3 segments a target image given a reference image and its mask, using a single frozen DINOv3 backbone, with no fine-tuning, no segmentation decoder and no auxiliary models. The paper's central claim goes past beating trained segmenters: fixing a positional bias in DINOv3's features improves those features beyond segmentation.

**visinf/INSID3** — [CVPR 2026 Oral] "INSID3: Training-Free In-Context Segmentation with DINOv3"

- Repository: https://github.com/visinf/INSID3
- Website: https://visinf.github.io/INSID3/
- Stars: 784 · Forks: 69
- Language: Python
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/visinf-insid3

## What the method does: segment from a reference pair, with the backbone frozen

INSID3 solves in-context segmentation entirely within a single frozen DINOv3 backbone. The phrase in-context is the key to understanding the task, and the minimal usage example makes it concrete.

You give the model a reference image, a reference mask, and a target image. The reference pair shows what the thing you want looks like, and the model produces a mask of the corresponding thing in the target. No training on your data, no fine-tuning of the backbone, no segmentation decoder head, no auxiliary models. Those four exclusions are stated in the feature list as the defining property of the method, and they are what make the approach distinctive.

The API is deliberately small. You build the model with build_insid3(), set the reference with model.set_reference(ref_image_path, ref_mask_path), set the target with model.set_target(target_image_path), and call model.segment() to get the predicted mask. Then a helper from utils.visualization draws the result.

The project came out of a CVPR 2026 Oral selection, with the paper and code released on 2026-03-29, the Colab demo on 2026-04-16, the oral selection announced on 2026-04-09, and semantic correspondence inference support added on 2026-04-21. The authors are from Politecnico di Torino, TU Darmstadt, TU Munich, hessian.AI, ELIZA and MCML.

The licence is Apache-2.0, the project page is visinf.github.io/INSID3, and the paper is on arXiv.

## The positional bias fix, which is the part that outlives segmentation

The most interesting claim in the README is not the benchmark result. It is that the authors uncover and fix a positional bias in DINOv3 features, improving their reliability beyond segmentation.

That is a bigger claim than a segmentation improvement, because it says the fix belongs to the backbone rather than to the task. If a frozen DINOv3 encoder carries a systematic positional bias, then that bias is present in every downstream use of those features, and correcting it inside INSID3 would produce better results for anyone else who adopts the same correction.

The mechanism implied by the name and by the rest of the repository is that the bias is corrected at inference time, since nothing is trained. The hyperparameter list confirms there is a positional component to the method: --tau, --merge-thresh and --svd-comps are named as hyperparameters with default values as in the paper, and they can be overridden from the command line, with the defaults living in opts.py. --svd-comps in particular suggests a singular value decomposition is part of how the representation is manipulated, which is a plausible route to removing a positional component from feature vectors.

The semantic correspondence section is where the broader claim gets tested. The authors analyze the effect of the positional debiasing on semantic correspondence using SPair-71k, which is a task about matching corresponding points between two images of the same object, not about producing masks. If debiasing helps there too, the claim that the fix is not segmentation-specific has some support.

That is a good structure for a research contribution: a method, and then a demonstration that the insight behind the method outlives the method.

## Setting up the environment, and the two routes with different assumptions

There are two supported setup routes, and they differ in more than package manager.

With Conda, the reference configuration is PyTorch 2.7.1 with CUDA 12.6, and the paper's experiments were run on that:

```bash
conda create --name insid3 python=3.10 -y
conda activate insid3
pip install -r requirements.txt
```

The optional CRF-based mask refinement is a separate git clone of a third-party CRF repository followed by a setup.py install, and it is not part of the base install.

With uv, the setup is faster and the CRF dependency is already included, which is the main practical difference between the two routes. The install script and the sync are:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

followed by uv sync and activating the .venv. The comment is explicit that CUDA 12.6 must be loaded beforehand, and that the sync creates the virtual environment and installs dependencies from pyproject.toml.

The pyproject.toml pins requires-python to ==3.10.*, which is stricter than most projects use, and the torch pin is exact at 2.7.1 with torchvision at 0.22.1. The uv configuration also restricts the environments to sys_platform == 'linux' and platform_machine == 'x86_64', so the uv route is not the portable one. The torch index is an explicit pytorch-cu126 index.

The requirements.txt is looser than the pyproject, with torch>=2.7 and numpy>=2.0, but it points at the same CUDA 12.6 wheel index.

## The DINOv3 weights are not bundled, and the default backbone is Large

The one manual step that will stop you is the weights, and the documentation is clear about it. INSID3 relies on a frozen DINOv3 backbone, and the pretrained weights have to be downloaded from the official facebookresearch/dinov3 repository. This is the correct arrangement: a third-party backbone is a dependency, and shipping someone's pretrained checkpoint inside your own repository creates licensing and provenance questions.

The instructions are to create a pretrain directory and place the backbone weights there:

```bash
mkdir -p pretrain
```

Three model files are named, a Large, a Base and a Small variant, all with the lvd1689m suffix and distinct hash suffixes. By default the Large model is used.

That default is worth a second look, because the whole point of the method is that it is training-free and cheap. The Large ViT with patch size 16 at a default input resolution of 1024 is a heavy forward pass to run for every image, and the documentation acknowledges the trade-off directly. For faster inference, reduce the input image size, with a smaller value substantially increasing speed with a minor performance drop, and the example given is 768.

So the practical shape is: one large frozen backbone forward pass per target image, plus a per-image cost that you can trade against resolution. The --model-size argument makes the same choice available from the command line, with small, base and large as the options and large as the default, so you can run the Small or Base backbone and see what it costs you in accuracy.

## Nine datasets and the fold structure behind the evaluation flags

The evaluation script covers a wide range of segmentation benchmarks, and the breadth is part of the generalisation claim.

The supported datasets are listed as coco, lvis, pascal_part, paco_part, isaid, isic, lung, suim and permis. That list spans natural imagery, part-level segmentation, remote sensing, skin lesions, lung imaging, underwater imagery and a permutation dataset, which matches the claim that the method generalises across natural, medical, underwater and aerial domains.

Running the evaluation is one command:

```bash
python inference_segmentation.py --dataset coco --exp-name insid3-coco
```

The flags that matter most are --model-size for the backbone, --shots for the number of reference images per episode with a default of 1, --image-size for the input resolution with a default of 1024, and --fold for the dataset fold with a default of 0. The fold structure is uneven across the datasets and the documentation is upfront about it: COCO, PACO-Part and PASCAL-Part have 4 folds, iSAID has 3, and LVIS has 10.

The --shots flag is the interesting one for understanding the method. With a default of 1, the method is evaluated in a one-shot setting, meaning a single reference image and mask per episode, which is the format that makes the in-context claim concrete rather than rhetorical.

The other flags, --tau, --merge-thresh and --svd-comps, are described as hyperparameters with default values as in the paper, and they are the knobs you would turn if you wanted to reproduce a specific result rather than use the defaults.

## Mask upsampling, optional CRF refinement, and the two-headed CLI

Two details of the output path affect how you should evaluate results.

By default the predicted mask is upsampled to the original image resolution using bilinear interpolation. Bilinear interpolation on a binary mask produces soft edges, so a prediction that looks slightly blurry at the boundary is the expected default behaviour rather than a bug.

For additional refinement there is CRF-based refinement, enabled in the library with mask_refiner="crf" when building the model and with --crf-mask-refinement on the command line. The CRF resolution is controlled with --crf-size and defaults to 640, and higher values may improve quality at the cost of speed. The CRF implementation is a third-party repository installed separately under the Conda route, and it appears in pyproject as a git dependency with setuptools and torch as extra build dependencies, which is why the uv route includes it automatically.

The repository layout reflects the two entry points. There is inference_segmentation.py and inference_keypoint_matching.py, plus models/, utils/, datasets/, assets/ and docs/. The keypoint matching script is what the semantic correspondence analysis uses, and it shares the same model and options machinery as the segmentation script.

Dataset preparation is documented separately, in docs/data.md, and the data/ directory at the root holds the dataset handling code. opts.py holds the option defaults, which is where you would look to find out what the paper's actual hyperparameter values are rather than guessing them from the command line help.

## Conclusion

Adopt INSID3 if you need to segment new categories without training data and without a GPU budget for fine-tuning, since the reference-image plus mask protocol means one example per class is enough to try. Do not adopt it expecting a stable, packaged model, because the repository has no releases, the uv configuration is restricted to linux on x86_64, and the CRF refinement path pulls a third-party repository as a build dependency. Before you run it, download the DINOv3 weights yourself from the official repository, create the pretrain directory and place the backbone file there, and expect to be on Python 3.10 with PyTorch 2.7.1 and CUDA 12.6.

## FAQ

### What do I need to run INSID3 for the first time?

Python 3.10 exactly, PyTorch 2.7.1 with CUDA 12.6, and the DINOv3 pretrained weights downloaded from the official facebookresearch/dinov3 repository and placed in a pretrain directory you create. The Large backbone is used by default.

### Does INSID3 need training or fine-tuning?

No. It is training-free, with no fine-tuning, no segmentation decoder and no auxiliary models. The DINOv3 backbone stays frozen and the method works from a reference image and its mask.

### How do I make INSID3 run faster?

Reduce the input image size, with a smaller value substantially increasing speed with a minor performance drop, for example building the model with image_size=768 against the default of 1024, or passing --image-size 768 to the inference script. You can also select a smaller backbone with --model-size.

### What is the positional bias fix in INSID3 about?

The authors report uncovering and fixing a positional bias in DINOv3 features, and say the fix improves their reliability beyond segmentation. They analyze the effect of the debiasing on semantic correspondence using SPair-71k, and the related hyperparameters include --tau, --merge-thresh and --svd-comps, with defaults in opts.py.

## Sources

- [Issues](https://github.com/visinf/INSID3/issues)
- [License: Apache-2.0](https://github.com/visinf/INSID3/blob/main/LICENSE)
- [Project website](https://visinf.github.io/INSID3/)
- [README](https://github.com/visinf/INSID3/blob/main/README.md)
- [visinf/INSID3 on GitHub](https://github.com/visinf/INSID3)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/visinf-insid3