SAMs-CDConcepts-Eval and the benchmark for context-dependent segmentation

What the paper set out to measure

The repository is the code release for a paper titled Inspiring the Next Generation of Segment Anything Models, published as arXiv 2412.01240 and cited in the README as Eva_SAMs with a 2025 year.

The argument the paper starts from is narrow and precise. SAM and SAM 2 are large-scale foundation models trained on billions of image and mask pairs covering a vast diversity of scenes, objects and contexts, and they have demonstrated excellent performance on segmenting context-independent concepts like people, cars and roads. What they overlook, in the authors' assessment, are context-dependent concepts: visual saliency, camouflage, industrial defects and medical lesions.

The reason is stated in terms of what the concepts require. Context-dependent concepts rely heavily on global and local contextual information, which makes them susceptible to shifts in different contexts and demands strong discriminative capability from the model. A model that learned to segment a car from its silhouette has learned something that transfers; a model that has to find a camouflaged object has to reason about the scene around it.

The stated motivation for the evaluation is that a lack of comprehensive evaluation of SAMs limits understanding of their performance boundaries, which may hinder the design of future models. The paper's stated aim is to provide insight that can inform the development of the next version, SAM 3.

So the audience is not application engineers looking for a segmentation model. It is researchers who need to know where a promptable segmentation foundation model stands, and where it does not, with a controlled setup rather than an anecdote.

The last push to the repository was on 2026-08-30 and it is not archived, though it has no GitHub releases and the licence is not stated in the repository listing, so treat reuse of the code itself as unresolved until you check the arXiv paper's terms.

Eleven concepts, and a unified prompting framework

The evaluation covers 11 context-dependent concepts across 2D and 3D images and videos, in various visual modalities, within natural, medical and industrial scenes.

The most substantial piece of engineering is the unified evaluation framework for SAM and SAM 2 that supports manual, automatic and intermediate self-prompting, aided by specific prompt generation and interaction strategies.

That three-way split is the interesting design decision. Manual prompting is the obvious case, a human clicking a point or drawing a box. Automatic prompting is the opposite extreme, the model prompted from information it extracts itself. Intermediate self-prompting sits between the two and is the part that is hard to define cleanly, because the prompt has to be generated by the model from its own earlier output rather than by a person.

Separating the three is what makes the results readable. A segmentation model that scores well under manual prompts has not demonstrated that it can find the object, only that it can follow a pointer once given one. If the paper reports results per prompting mode, then a weak automatic score is a statement about the model's own grounding rather than about the mask decoder, and that distinction only exists because the framework keeps the modes apart.

Two further strands are named in the paper summary. The authors explore the potential of SAM 2 for in-context learning, and they add a test of how prompts behave when they are imperfect, simulating real-world usage. The second is arguably the more useful for practitioners, because nobody in production supplies a perfect click at the centre of the object; they supply an approximate click, a loose box, or a text description, and a model that only works with ideal inputs is not usable.

The repository layout follows the same split. There are run_sams_for_image.py, run_sam2_for_video.py, run_sam1_for_video.py and run_sam2_for_incontextlearning.py, so each task has its own entry point rather than one script with a mode flag, plus a utils/ directory and a preprocess/ directory for dataset conversion.

Preparing SAM and SAM 2 for the evaluation

The setup instructions install the two upstream models separately, which matters because the README explicitly says to use a separate folder for the sam2 code.

SAM goes in first:

git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

SAM 2 follows the same shape, into a different directory:

git clone https://github.com/facebookresearch/sam2.git segment-anything2
cd segment-anything2
pip install -e .

Then the checkpoints. The README asks you to download both and assign their paths to the items sam-l and sam2-l of the config file. For SAM that is the vit_l checkpoint, whose url is https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth. For SAM 2 it is the hiera_large checkpoint, whose url is https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt.

Both are the large variants, which is the relevant choice for reproducing published numbers. A smaller checkpoint would run faster and change the results, and since the config carries the paths as named items, swapping checkpoints is a config edit rather than a code change.

Note what is not here. There is no requirements file with the evaluation dependencies, no environment specification and no pinned versions for the upstream models. The pyproject.toml in the repository holds formatting configuration for the project's own code, with isort set to the black profile, black and ruff both at a line length of 119, and ruff's format section set to double quotes and space indentation. That is a style contract, not a dependency lock.

The practical implication is that you are assembling the environment yourself, from two upstream repositories, and that reproducibility will depend on your own pinning discipline.

Datasets: not bundled, and not uniform

The dataset section is short and points outward. It says to see the arXiv version for dataset details and to set their paths in the config file.

So the repository does not ship the data. The evaluation depends on the exact datasets the paper used, which the reader has to obtain from the sources described in the paper, and the config file is where the paths are set.

The note attached to that section is where the practical trap is. Some datasets need to be preprocessed before use by the scripts in the preprocess folder. And the note warns that images in some datasets, like CAD for Video COD, may not have corresponding annotations, and that these images will not be used for model prediction and performance evaluation, so they should be cleaned up in advance before the script is used.

That is a specific and consequential instruction. If unannotated images are left in place, the pipeline silently excludes them, which means the evaluation covers a different set of inputs than you think it does. Any table you then produce is not comparable to the published numbers, and the difference is invisible unless you check the annotations yourself first.

The config.yaml file at the repository root is the junction between the data and the runs, and it holds at least the two checkpoint path items, sam-l and sam2-l. The run.sh script at the root is the entry point the README points to for generating predictions, with the instruction to run the corresponding commands to generate predictions for each task, so the per-task commands live in that script rather than in the README.

The consequence for adoption is that this is a research harness with a data dependency, not a packaged benchmark. Reproducing a specific number from the paper means matching the paper's dataset versions, its preprocessing and its checkpoint choices, and the repository gives you the machinery for all three without pinning any of them for you.

Measurement tools and the surrounding research code

The repository does not implement its own segmentation metrics. The Evaluation Tools section points to two external projects: PySegMetric_EvalToolkit and PyADMetric_EvalToolkit.

The split between them suggests the kinds of masks being scored. A general segmentation metric toolkit handles region overlap measures, and a separate anomaly-detection metric toolkit handles the industrial defect case, where the question is whether a mask finds a defect that occupies a small part of the image, and IoU against a tiny ground truth is a poor way to express that.

This delegation is worth noting for anyone reproducing the work. The metrics live in other repositories with their own versions and their own conventions, so a discrepancy between your numbers and the paper's may come from the metric implementation rather than from the prediction pipeline.

The repository also links to a related curated list, Awesome-Unified-Context-dependent-Concept-Segmentation, which collects related work on unified segmentation of context-dependent concepts. The contributors are credited individually in the README as @lartpang, @Xiaoqi-Zhao-DLUT, @DUT-CSJ and @yuanzhao-CVLAB, with the paper's author list being longer still: Xiaoqi Zhao, Youwei Pang, Shijie Chang, Yuan Zhao, Lihe Zhang, Chenyang Yu, Hanqi Liu, Jiaming Zuo, Jinsong Ouyang, Weisi Lin, Georges El Fakhri, Huchuan Lu and Xiaofeng Liu.

The repository root is small and purposeful: config.yaml, run.sh, the four run scripts, preprocess/, utils/, pyproject.toml, .gitignore and README.md. There is no test directory and no CI configuration in the listing, so unlike a library, this code does not carry its own regression suite. That is normal for a research release, but it means the scripts are a reference implementation rather than a maintained tool.

What this benchmark will not tell you

Three boundaries are worth stating before anyone builds on this.

The first is scope. The evaluation covers SAM and SAM 2 only. The scripts are named run_sams_for_image.py, run_sam1_for_video.py and run_sam2_for_video.py, and the checkpoint configuration has slots for exactly two models. If you want to know how a different segmentation foundation model performs on context-dependent concepts, this repository does not give you the harness for it. The awesome list of related work is where a broader survey would start, but the code is specific.

The second is the model scale. Only the vit_l checkpoint for SAM and the hiera_large checkpoint for SAM 2 are described. Both are the large variants. Nothing in the README indicates that the smaller checkpoints were evaluated, so the results describe capability at the top of the scale, and the cost of running the evaluation follows from that.

The third is the documentation gap. There is no environment lock, no pinned dataset versions, no per-task command list in the README, and no test suite in the repository listing. The arXiv paper is the specification, and the README explicitly redirects the reader to it for dataset details. Anyone planning to extend the framework, add a concept family or a new prompting mode will be reading the paper and the upstream model repositories, then reverse-engineering the plumbing from run.sh.

None of that is unusual for a research release, and the paper is a serious artefact from a group with a track record in the area. But the honest framing is that this is the authors' evaluation code, published so others can read it, cite it and build on it, not a supported tool with a compatibility promise.

Editorial conclusion

SAMs-CDConcepts-Eval is a good fit for a researcher who needs to measure where a promptable segmentation model actually breaks, because it separates the prompt type from the concept difficulty and covers eleven concept families across natural, medical and industrial scenes rather than the usual people-and-cars test sets. It is a poor fit if you want a turnkey benchmark, since the datasets are not included and the README sends you to the arXiv paper for details and to the preprocess scripts for conversion, and a poor fit if you want to run it against a model other than SAM or SAM 2, since the scripts are named for those two specifically. Before running anything, read the dataset preparation notes in the arXiv version, because some datasets need preprocessing and some, such as CAD for Video COD, contain images with no corresponding annotations that have to be removed beforehand or the evaluation will be wrong.

Frequently asked questions

Official sources

1. Issues
2. lartpang/SAMs-CDConcepts-Eval on GitHub
3. Project website
4. README