InstructSAM: An Instruction-Driven Multi-Instance Segmenter Built on Qwen3-VL

What InstructSAM Actually Solves

Most segmentation models are locked to a single prompt type. You either give a category name, a referring expression, or a click point, and you get one mask or a semantic region. InstructSAM is built for the case where the instruction is free-form natural language, and the expected output is a set of instance masks, not a single region. The README lists three instruction types: category prompts, referring expressions, and reasoning-style instructions. The reasoning style is the interesting one. It means the model can handle a sentence like "find the person holding the umbrella" without a separate object detector or a multi-step agent. The intended audience is researchers working on instruction-based segmentation, particularly those who want to train or evaluate on the new Inst2Seg dataset. The project is not a lightweight demo. It is a training framework with two stages, and it expects you to download raw images from original dataset sources.

How the Two-Stage Training Works

The training pipeline starts from the Qwen3-VL-2B-Instruct checkpoint. Stage 1 is a standard fine-tuning run, launched with bash scripts/train/stage1.sh. After that, you must merge the Stage 1 LoRA weights back into the base model using a dedicated command. The merge step is explicit: you give a base directory, the stage 1 model path, and a save path. Stage 2 then starts from that merged checkpoint, not from the original base. This two-stage design is common for instruction tuning, where the first stage teaches the model the segmentation task and the second stage adds reasoning capabilities. What is notable is that the merge is not automatic. The README gives the exact command, so it is a deliberate step, not an afterthought. The training data consists of annotation JSON files downloaded from Hugging Face, placed under data/training. Raw images come from the official source of each dataset, which means you need to handle licensing and download logistics yourself. The repository does not bundle images, only annotations.

Inference: A Single Command, But With Caveats

For inference, the README shows a single command: python3 -m instructsam.infer --model_path work_dirs/stage2 --image-path path/to/image.jpg --query "Please segment the object in the image." --output-dir vis. The script prints generated text and mask scores, then writes overlays to vis/. That is straightforward. But the model path in the example points to work_dirs/stage2, which implies you have trained the model yourself. There is a Hugging Face link for a pretrained InstructSAM-2B model, but the README does not explicitly say that the inference script accepts a Hugging Face hub path. It likely does, but that is not confirmed. Also, the evaluation section warns that Python evaluation files require explicit --image_folder and --question_file arguments instead of machine-specific defaults. That suggests the codebase was written with hardcoded paths in places, and you must override them. The inference command itself does not show such arguments, so it may work with relative paths, but the evaluation scripts definitely need manual configuration.

The Inst2Seg Dataset and Evaluation Scripts

InstructSAM introduces its own dataset, Inst2Seg, with a separate benchmark split called Inst2Seg-Bench. The training and evaluation annotation JSON files are hosted on Hugging Face, and the README provides four evaluation entry points: eval_inst2seg.sh, eval_reasonseg.sh, eval_grefcoco_ap.sh, and eval_roborefit.sh. The first is for the native dataset, and the others are for external benchmarks: ReasonSeg, gRefCOCO with average precision, and RoboRefIt. That coverage is a strong point. Most instruction segmentation papers only evaluate on one or two datasets. Here you get four, and they span referring expression and reasoning tasks. But the README is thin on how to prepare those external datasets. It says to edit dataset roots in each script, but it does not list the expected directory structure or the format of the question files. The evaluation scripts are there, but you have to reverse-engineer the data layout from the Python code. That is a real cost for anyone who wants to reproduce the paper's numbers.

A Native C++ Port for CPU-Only Inference

The README points to a third-party project, instructSAM.cpp, which ports the model to native C++ with ggml for CPU-only inference. The claim is that a Docker one-liner runs end-to-end in about 1.7 minutes on a 12-core laptop, and the weights are converted to GGUF at build time. This is a significant practical option because the primary Python inference path likely requires a GPU, given the use of flash-attn in the setup. The C++ port does not republish the weights; it converts them locally. That is a useful workaround for someone who wants to test the model on a laptop without a GPU. However, the README does not give the exact Docker command, only a pointer to the external repository. So you have to visit that repo for the one-liner. Also, the port is a third-party contribution, not an official part of InstructSAM, so its fidelity to the original model is not guaranteed by the main project.

Limitations and Missing Documentation

The most obvious limitation is the unknown license. The repository metadata says license: unknown, and the README does not mention a license. That is a blocker for commercial use and even for some academic projects that require a clear license for redistribution. The second limitation is the dependency on flash-attn. The setup command pip install flash-attn --no-build-isolation is known to be finicky on many systems, and it often requires a specific CUDA version. If you cannot install flash-attn, the model may not run at all. Third, the README does not specify the exact format of the annotation JSON files, the image directory structure, or the expected output of the inference script beyond mask overlays. The evaluation scripts are present but require manual path editing, and the README warns about machine-specific defaults. That means the codebase is not polished for out-of-the-box use. For someone who wants to quickly try the model, the C++ port might be easier, but for training or evaluation, you need to be comfortable reading Python code and fixing paths.

Alternatives and How They Differ

The obvious alternative is Grounding DINO combined with SAM, which is the classic two-stage pipeline. You use Grounding DINO to detect boxes from text, then SAM to segment each box. That approach requires two models and an explicit box generation step, and it does not handle reasoning-style instructions well because the text encoder is not a large language model. InstructSAM instead uses a single Qwen3-VL backbone that directly outputs instance masks, which avoids the multi-round agentic prompting that some vision-language models require. Another alternative is SEEM, which also handles multiple prompt types, but it is older and does not have the reasoning capabilities of a modern LLM-based backbone. The key difference is that InstructSAM is built on Qwen3-VL, so it inherits the language understanding of a 2B-parameter LLM, while Grounding DINO uses a smaller text encoder. That matters for complex instructions. But the trade-off is that InstructSAM is a heavier model and requires a two-stage training recipe, whereas Grounding DINO+SAM can be used off-the-shelf with no training.

Maintenance and Upgrade Cost

The repository has no recent releases and no release history. The last push date is unknown, and the project is not archived, so it may be actively maintained, but there is no evidence in the README. The citation is for a 2026 arXiv paper, which suggests the project is new. The maintenance cost is twofold. First, you must keep up with the Qwen3-VL base model updates if you want to retrain, because the LoRA merge step is tied to a specific base checkpoint. Second, the evaluation scripts reference external datasets that may change their formats over time. The README does not provide a changelog or migration notes. The license being unknown also adds legal overhead if you plan to distribute any modified version. For a research project, the cost is manageable if you are comfortable with the setup. For a production deployment, the lack of a license and the flash-attn dependency are serious barriers.

Editorial conclusion

Adopt InstructSAM if you need a single model that handles category prompts, referring expressions, and reasoning-style instructions while outputting multiple instance masks, and if you can tolerate a two-stage training pipeline with manual LoRA merging. Do not use it if you require a plug-and-play inference package without dataset setup, or if you need a permissively licensed codebase, since the license is listed as unknown. Before committing, verify that the Hugging Face model and dataset links resolve, check the exact format of the evaluation JSON files, and test the inference script on your own image to confirm the output overlays match your expectations.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. README
4. Releases