Open-Vocabulary SAM moves CLIP knowledge in both directions

Two modules, two directions of transfer

The model is described as SAM-inspired and designed for simultaneous interactive segmentation and recognition. The mechanism is two unique knowledge transfer modules. SAM2CLIP adapts SAM's knowledge into CLIP through distillation and learnable transformer adapters, and CLIP2SAM transfers CLIP knowledge into SAM to enhance its recognition capabilities. The bidirectionality is the point: a promptable segmenter like SAM knows where things are but not what they are, and CLIP knows what things are but is not a segmenter, so the design moves information along both edges rather than bolting a classifier onto the end. The paper title itself states the goal as segment and recognize twenty thousand classes interactively, which is the combination that makes the model interesting for open vocabulary work rather than a fixed-class detector.

The install is an OpenMMLab stack pinned to one commit

The environment is managed with conda, and the installation is a sequence of steps that has to be followed in order. PyTorch comes from the pytorch channel with a CUDA 12.1 build, mmengine is a specific release archive, and mmcv is installed from a git commit rather than a release, with the README warning that older versions before that commit may cause bugs:

TORCH_CUDA_ARCH_LIST="{COMCAP}" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" CUDA_HOME=$(dirname $(dirname $(which nvcc))) LD_LIBRARY_PATH=$(dirname $(dirname $(which nvcc)))/lib MMCV_WITH_OPS=1 FORCE_CUDA=1 python -m pip install git+https://github.com/open-mmlab/mmcv.git@4f65f91db6502d990ce2ee5de0337441fb69dd10

That single line is the sharpest edge in the project. The compute capability value has to be filled in by hand, and the README's advice for obtaining it is to ask a language model the question of what the compute capability of your GPU model is, with the instruction to output only the number. The rest of the environment is a set of additional OpenMMLab packages (mmdetection, mmsegmentation, mmpretrain), the panoptic API and the LVIS API forks, and a list of extra packages including timm, kornia and scikit-image.

Training runs through a distributed launcher script

Every training and inference step goes through the same entry point, `tools/dist.sh`, with a config path and a GPU count. The order matters because the language embeddings have to exist before the model is trained. First the language embeddings are extracted with the `gen_cls` command on the ovsam config, then SAM features are dumped with the sam2clip config, then SAM2CLIP is trained, then CLIP2SAM is trained, and finally inference runs the same ovsam config in test mode:

bash tools/dist.sh gen_cls seg/configs/ovsam/ovsam_coco_rn50x16_point.py 8
bash tools/dist.sh test seg/configs/sam2clip/sam_vith_dump.py 8
bash tools/dist.sh train seg/configs/sam2clip/sam2clip_vith_rn50x16.py 8
bash tools/dist.sh train seg/configs/clip2sam/clip2sam_coco_rn50x16.py 8
bash tools/dist.sh test seg/configs/ovsam/ovsam_coco_rn50x16_point.py 8

The config names encode the backbones, with a ViT-H SAM and an RN50 with 16x feature maps for the main ovsam config, and a CLIP2SAM config with the same RN50x16 backbone. The `8` at the end is the number of processes, so the whole pipeline assumes a multi-GPU machine, and there is no single-GPU variant documented.

Datasets are laid out the way mmdetection expects

The data preparation follows the mmdetection convention, with datasets placed in a `data/` folder in the project. The COCO layout needs an `annotations` directory holding the panoptic and instance JSON files for train and val, the `train2017` and `val2017` image directories, and a `panoptic_{train,val}2017/` directory of PNG annotations. The SAM layout is different in kind, since it is a list of image folders rather than a fixed directory structure, and each sample directory holds a JPEG and a JSON. The split files `train.txt` and `val.txt` contain the names of the folders to use, one per line. That indirection is worth understanding before you start, because it means the SAM split is a text list you edit rather than a directory convention the loader discovers.

Weights come from Hugging Face, code comes from GitHub

The pre-trained weights are not in this repository. The README points to a Hugging Face model collection, ovsam_models, and gives a command to fetch it into a directory named `models`:

git clone https://huggingface.co/HarborYuan/ovsam_models models

The same hosting is used for the demo. There is an online demo on a Hugging Face Space, and the instructions for running the gradio demo locally are to clone that Space repository, create a conda environment on Python 3.10, install a pinned gradio 4.7.1, install the requirements file and run `python main.py`. The project page is hosted at the MMLab NTU site, and there is also an OpenXLab listing. The distribution of the demo through Hugging Face Spaces rather than only through the GitHub repository is a practical choice, because the weights are large and a Space already has a working environment around them.

The RWKV-SAM follow-up lives in a subfolder

A second line of work is included in the same repository, under `projects/rwkvsam`, with its own readme for the details. RWKV-SAM is a separate paper with its own author list, which includes researchers from S-Lab at Nanyang Technological University, SkyworkAI and UC Merced, and its package description is RWKV-SAM: RWKV Augmented Segment Anything Model. The framing in the paper title is a question about architecture choices, Mamba or RWKV, and the repository includes a `projects/` directory alongside `seg/` and `ext/`, with the package manifest including all three. So the repository is two projects in one tree, which is convenient for shared tooling such as the distributed launcher and the dataset preparation, and slightly awkward if you only want one of them.

The test configuration is stricter than most research repositories

The pyproject file gives a sense of how the project treats its own tests. The build backend is poetry, the coverage configuration sets `fail_under = 80`, so a coverage run below eighty percent fails, with branch coverage enabled and the test and config files omitted from the report. The pytest configuration treats warnings as errors, with only `UserWarning` and `DeprecationWarning` ignored, and runs with `--strict-markers` and `--strict-config`, so an unregistered marker is an error rather than a warning. Three markers are declared, for unit, integration and slow tests. There is also a `poetry.lock` file at the root, which means the dependency versions are pinned. The odd detail is the package name, which is `rwkvsam` at version 0.1.0 with placeholder author fields, so the packaging metadata still reflects the RWKV-SAM side of the repository rather than the Open-Vocabulary SAM project as a whole.

Editorial conclusion

Adopt this repository if you are reproducing or extending the Open-Vocabulary SAM work, since it carries the training and inference configs for both transfer modules and the RWKV-SAM follow-up in one tree. Do not adopt it as a drop-in segmentation library, because the installation requires a specific mmcv commit, a manual compute capability value and an OpenMMLab toolchain that has to be assembled in the right order. Verify first that your GPU's compute capability matches what you pass to the mmcv build, since the README asks you to obtain that number yourself, and note that the pre-trained weights live on Hugging Face rather than in this repository.

Frequently asked questions

Official sources

1. HarborYuan/ovsam on GitHub
2. Issues
3. Project website
4. README