Sa2VA: A Pixel-LLM Repository Where SAM-2 Meets Multimodal LLMs

The gap Sa2VA is built to close

Most multimodal LLMs describe a scene in words. A referring segmentation model returns a mask. Getting both from one system is the problem this repository addresses, and the README states the goal directly: a unified model that marries SAM-2 with MLLMs for dense grounded understanding of images and videos. The listed capabilities are referring segmentation, grounded conversation, visual prompting, and image and video chat. That combination is what the project calls pixel-level grounding, and it is the thread connecting every project in the repository. The audience is research groups and applied teams working on segmentation, grounding, and video object tracking who want a single backbone to handle language and masks rather than stitching two models together. The README also names the supported backbones: InternVL2.5 and InternVL3, plus Qwen2.5-VL and Qwen3-VL. That list is the clearest signal of who this is for. If you are already running one of those MLLMs, Sa2VA is positioned as the segmentation layer on top of it.

One repository, four papers, one environment

The layout is a monorepo. Sa2VA itself lives under projects/sa2va and is described as the core unified model. Three siblings sit beside it. VRT, Visual Reasoning Tracer, is object-level grounded reasoning built on Sa2VA and ships two artifacts: VRT-Bench for evaluation and VRT-80k for training data. SAMTok is a mask-token interface that the README says lets any MLLM generate and understand masks, summarized by its paper title as representing any mask with two words. SaSaSa2VA is a segmentation-augmented extension that the README credits with first place in the ICCV 2025 LSVOS Challenge RVOS Track. Pixel-SAIL is listed as a single-transformer approach to pixel-level grounding. The dependency direction matters: Sa2VA is the anchor, and VRT and SaSaSa2VA are built on top of it. SAMTok is a different interface idea, replacing mask generation with a token vocabulary. If you only need referring segmentation, most of this repository is not for you, and the shared environment means you still carry its dependency footprint.

How the environment is pinned

The repository manages dependencies with uv, and the README is explicit about why: dependencies are declared in pyproject.toml and every transitive package is version-locked in uv.lock. Install uv once with the documented shell command, then either run the helper script at the repository root or sync manually from projects/sa2va. The helper places the virtualenv in /tmp and symlinks it back into the project. Two extras exist. The default is latest, which the README associates with InternVL3 and Qwen3-VL era backbones. The alternative is legacy, for InternVL2.5 or earlier. That split is the single most important decision before your first sync, because the README presents it as a choice between two dependency sets rather than a runtime flag. Tokens for HuggingFace and OpenRouter come from a .env file copied from .env.example, which setup_env.sh loads automatically. Training and evaluation are run from the repository root with the environment activated, and each project README carries its own steps.

What the repository does not tell you

The root README is a directory, not a manual. It does not list GPU memory requirements, does not give expected training times, and does not state which checkpoint corresponds to which backbone. For those you have to open the per-project README under projects/sa2va or the model zoo collection linked from the project entry. The uv.lock file pins a full transitive dependency set, which buys reproducibility but also means upgrading one package in isolation is not the intended workflow. The README does not describe a compatibility policy across the four projects, so it is not clear whether a change made for SAMTok can break the Sa2VA training path. The repository is also multi-paper: Sa2VA is tagged TPAMI 2026, SAMTok CVPR 2026, and VRT an arXiv preprint from 2025. Papers move at a different pace than code, and nothing in the supplied material indicates how the code tracks the published versions.

Where the monorepo shape costs you

A shared environment across four research projects is a deliberate trade. You get one uv sync instead of four, and the README frames that as the point: a single source of truth recreated exactly with one command. The cost is that you cannot install Sa2VA alone at a smaller dependency weight, and you cannot pin a different version of a shared package for one project without leaving the documented path. The legacy and latest extras mitigate this at the backbone level, but only at that level. There is a second cost that is easy to miss. The README says training and evaluation run from the repository root while the environment lives under projects/sa2va. That split between working directory and environment location is the kind of detail that produces confusing import errors if you skip the activation step, and nothing in the material suggests the code detects and corrects it for you.

How this differs from a segmentation toolkit

The obvious comparison is a dedicated segmentation library such as the SAM family on its own. The difference is architectural, not a matter of accuracy. A segmentation toolkit takes a prompt and returns a mask. It has no language model, so it cannot hold a conversation about what it segmented, and it cannot answer a question whose answer depends on identifying an object first. Sa2VA puts the MLLM in the loop, which is what makes grounded conversation and visual prompting possible from one model. The reverse trade is equally real. A dedicated segmentation library is a smaller dependency surface with a narrower, more predictable interface, and if masks are all you need, the language model is overhead you are paying for in memory and setup complexity. SAMTok sits at a third point on that spectrum, replacing mask generation with a token interface that any MLLM can use, which is a different bet about where the interface should live.

Licence and the cost of keeping up

The repository is Apache-2.0, which permits commercial use and modification, but the licence covers the code in this repository and says nothing about the weights, datasets, or upstream backbones it depends on. VRT-80k training data and the model zoo collections are hosted separately on Hugging Face, and each of those carries its own terms that you need to check yourself. This is not legal advice; read the terms attached to each artifact before you ship anything. On maintenance, the material shows two releases, v1 and v2, roughly a day apart in October 2025, followed by a push in September 2026. That cadence suggests active development rather than a frozen release, which cuts both ways: fixes arrive, and so do changes you have to absorb. Because the environment is locked in uv.lock, the practical upgrade cost is re-syncing and re-checking the two extras against whichever backbone you run, not editing requirements by hand.

Editorial conclusion

Adopt Sa2VA if you are doing research or evaluation on referring segmentation, grounded conversation, or visual prompting and you can commit to the uv-managed environment described in the README. Do not adopt it if you need a stable, narrowly scoped inference library with a documented upgrade path, because the repository is a monorepo of four papers with a shared environment and the README does not describe a compatibility or deprecation policy. Before you start, verify three things: which extra matches your backbone (latest for InternVL3 and Qwen3-VL, legacy for InternVL2.5 or earlier), whether the Sa2VA model zoo collection on Hugging Face actually contains the checkpoint you plan to run, and whether the project README under projects/sa2va lists the training and evaluation entry points for your task.

Official sources

1. bytedance/Sa2VA on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases