InstructSAM

Code for InstructSAM, an instruction driven segmentation framework that takes category prompts, referring expressions, or reasoning style instructions and outputs a set of instances.

InstructSAM is an instruction driven multi instance segmentation framework, and the code corresponds to a paper on segmenting any instance with any instructions. Inputs can be category prompts, referring expressions, or reasoning style instructions, and the output is a predicted set of instances rather than a single mask. That flexibility is the core of what the framework is offering.

Training annotation JSONs are hosted on Hugging Face and go under the data training directory. Evaluation annotation JSONs are also on Hugging Face and belong in the data eval directory. Download and place them in the right folders, and the data side of the setup is done.

The framework advertises efficient inference that avoids multi round agentic prompting and repeated SAM calls. It also supports the Inst2Seg dataset, with training and evaluation scripts for instruction based instance segmentation, so the repo covers the full workflow rather than just the model.

Before evaluation, the dataset roots in each script need editing. The Python evaluation files take explicit image folder and question file arguments instead of relying on machine specific defaults, which keeps the evaluation reproducible across machines.

A third party contribution provides instructSAM as a native C++ ggml port for CPU only inference. A Docker one liner runs the whole thing end to end in about 1.7 minutes on a 12 core laptop and converts the InstructSAM 2B weights to GGUF at build time, which makes the model usable without a GPU.

• Source 1 · Project repository
• Source 2 · README