SAM-Adapter-PyTorch: training a task adapter on top of a frozen SAM backbone

Where SAM underperforms and the adapter takes over

SAM-Adapter exists because promptable segmentation degrades in scenes where the target has no clean edge contrast: a camouflaged animal against foliage, a shadow on tarmac, a polyp against mucosa. The paper behind the repository is an ICCV 2023 entry on pages 3367 to 3375, also on arXiv as 2304.09148, and its move is to keep the SAM backbone frozen while training a small task adapter with prompts on top of it. That reframes the cost question: you add adapter parameters, not a second copy of a ViT. The audience is narrow and readable off the dataset list the project ships with, COD10K, CAMO and CHAMELEON for camouflaged object detection, ISTD for shadow detection, Kvasir for polyp segmentation. An update dated 28 April 2023 added the polyp experiment, which is where the medical dataset came from. Reported results belong to the ViT-H backbone and to configs/demo.yaml; ViT-L and ViT-B configs exist for cards with less memory, at some cost to accuracy.

Two later branches swap the backbone, and what the code borrows

Two later branches abandon the frozen ViT-H starting point. The SAM2-Adapter branch, added in an 8 August 2024 update, adapts Segment Anything 2, which that update introduces as a more capable backbone, and it carries its own technical report, arXiv 2408.04579. The SAM3-Adapter branch, added 25 November 2025, does the same for Segment Anything 3. That is a different bet from an adapter on frozen weights: you inherit the quality of a newer backbone and accept its own interface. The other honest comparison sits closer to home, because the project states that part of its code derives from Explicit Visual Prompt by Weihuang Liu, Xi Shen, Chi-Man Pun and Xiaodong Cun of University of Macau and Tencent AI Lab. Explicit Visual Prompt also bolts learnable prompts onto a frozen SAM, but as a general visual-prompt method. SAM-Adapter commits to the underperformed scenes list and to per-dataset configs, which narrows the surface and keeps the failure modes inside the published scope. If a newer SAM checkpoint is already on disk, a branch is the shorter route; if the ICCV numbers are the target, the main branch is what matches.

Memory sets the boundary, and two training scripts share it

Memory, not code, sets the boundary here. The quick start warns that "the SAM model consume much memory", and says the training run used 4 x A100 graphics cards, with a note to move to cards with larger memory when you hit the wall. Two training entry points appear in the documentation and the difference between them is never stated: loadddptrain.py is the script the quick start launches with configs/demo.yaml, while train.py is offered separately under Train. Nothing in the repository entries explains what the load-ddp variant changes. Both commands run one node with four processes through torch.distributed.launch, so the script name is the only variable a reader can change, and choosing wrong costs memory rather than accuracy. On smaller cards the other lever is backbone size, since the ViT-L and ViT-B configs exist for that case.

The faster-training snippet repeats the line above it

The Train section offers a two line block that the surrounding text presents as a route to faster training, crediting a comment in issue #39, and the block is worth reading line by line. Its second line is the distributed launch command the quick start already gave, identical down to the four visible devices and the demo config. Its first line is the only genuinely new command, a bare torchrun invocation, and it opens with an exclamation mark, which a shell reads as the start of a word rather than as the notebook escape that makes such a line work in Jupyter or Colab. So the snippet changes no flag at all: the reader is choosing between two script names, one of which the page elsewhere presents as an ordinary shell command. Copying the block into a shell as printed dies on the first line, and the error has nothing to do with GPUs or checkpoints.

!torchrun train.py --config configs/demo.yaml

The command sequence from an empty directory to test.py

Environment first: the code was implemented with Python 3.8 and PyTorch 1.13.0, and one line installs the pinned packages.

pip install -r requirements.txt

Then two file placements: the dataset goes into ./load, and the pre-trained Segment Anything checkpoint, taken from Meta's own repository, goes into ./pretrained. Training uses the distributed launcher with four processes on one node.

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nnodes 1 --nproc_per_node 4 loadddptrain.py --config configs/demo.yaml

What to watch after that launch is the memory ceiling rather than the log. The Train section offers the same launcher against the other script, with the config as the trailing argument.

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch train.py --nnodes 1 --nproc_per_node 4 --config [CONFIG_PATH]

Evaluation is a two argument call, and no dataset layout under ./load beyond the folder name is documented, nor any annotation format.

python test.py --config [CONFIG_PATH] --model [MODEL_PATH]

A cu116 torch pin and a typing backport to inspect

requirements.txt is short and specific, and two entries deserve a second look before you install anything. The torch line is torch~=1.13.0+cu116 and the matching torchvision is 0.14.0+cu116, both cu116 wheels, while the documented install is a plain pip install -r requirements.txt with no wheel index named. Where those artifacts are expected to come from is not documented, and on a clean machine that is the first thing to settle. The second oddity is the typing backport pinned at 3.7.4.3, a package that installs a module named typing over the standard library one, which is a strange addition to a Python 3.8 environment that already carries it. The remainder is ordinary imaging and reporting: imageio 2.9.0, opencv-python 4.7.0.72, PyYAML 6.0, scipy 1.10.1, timm 0.3.2, plus thop and torchsummary for model inspection and onnxruntime 1.14.1. No SAM package is listed anywhere in the file, which is why the quick start sends you to a manual download instead of a dependency.

A duplicated zip name, a doubled citation key, one metric file

Scoring lives in sod_metric.py at the repository root, and that is the file to read first when a checkpoint's numbers will not line up with the paper. Weights are shared through one Google Drive link, and two PDFs, Adaptor.pdf and SAM2Adapter.pdf, sit beside the code. Three details reward a close look. The root entry SAM3-Adapter-Pytorch(2).zip carries a duplicated (2) suffix, so the archive the listing shows is not the name the project publishes, and SAM3 code is pointed at elsewhere as a branch rather than as that file. The citation block then defines the same BibTeX key chen2023sam twice, once as an inproceedings entry for the ICCV paper and once as a misc entry for the arXiv preprint, so a bibliography build drops one of the two in silence.

@inproceedings{chen2023sam,
@misc{chen2023sam,

The byline disagrees with its own entry as well, listing Papa Mao before Ying Zang in the author line while the BibTeX entry puts Papa Mao last. The licence is MIT, there are no GitHub releases to pin a version against, and the last push was on 2026-05-17.

Editorial conclusion

Adopt SAM-Adapter if your images are camouflaged objects, shadows or polyps and you can give the job four A100s plus a manually placed SAM checkpoint. Leave it alone for ordinary promptable segmentation, where plain Segment Anything already works, and sort out where cu116 wheels for torch 1.13.0 come from before the pip line, because the install command names no index. First check: run test.py against the shared checkpoint on your own images and read what sod_metric.py reports before trusting any config.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. tianrun-chen/SAM-Adapter-PyTorch on GitHub