# SAM2Long keeps several segmentation hypotheses alive so SAM 2 survives a long video > SAM2Long is the official implementation of an ICCV 2025 paper that fixes SAM 2's error accumulation on long videos by giving it a training-free memory tree, which keeps several candidate segmentations and prunes the weaker ones as the video runs. It plugs into the official SAM 2 setup rather than replacing it, and its own code is released under a non-commercial licence. **Mark12Ding/SAM2Long** — [ICCV 2025] SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree - Repository: https://github.com/Mark12Ding/SAM2Long - Website: https://mark12ding.github.io/project/SAM2Long/ - Stars: 574 · Forks: 21 - Language: Jupyter Notebook - License: NOASSERTION - Published: 2026-10-22 · Updated: 2026-10-22 · Language: en - Canonical page: https://hysenlabs.com/projects/mark12ding-sam2long ## The problem is error accumulation, not a weak per-frame model SAM 2 is a promptable segmentation model, and on short clips it is good enough that you would not look for a defect. The problem SAM2Long addresses shows up over time. The README describes SAM2Long as significantly improving on SAM 2 by addressing the error accumulation issue, particularly in long-term video scenarios involving object occlusion and reappearance. The segmentation process becomes more resilient and accurate over time, maintaining strong performance even as objects are occluded or reappear in the video stream. That framing matters for how you evaluate it. This is not a new backbone or a better prompt encoder. It is a change to how SAM 2's memory is managed once the video runs past the length the model was comfortable with. The model's per-frame output quality is unchanged; what changes is whether the tracker can recover after it loses the object. The reported gain is described as an average improvement of 3 in J and F scores across all 24 head-to-head comparisons on long-term video datasets such as SA-V and LVOS. Twenty-four comparisons is four backbones times three datasets times two checkpoint families, and the claim is that the improvement is consistent across all of them rather than concentrated in one configuration. The headline comparison table shows the shape of it. On SA-V validation with the SAM 2 checkpoints, the Tiny backbone goes from 73.5 to 77.0, Small from 73.0 to 77.7, Base+ from 75.4 to 78.4 and Large from 76.3 to 80.8. On SA-V test the gains are smaller for the larger backbones, where Large moves from 75.5 to 80.8 but Tiny moves from 74.6 to 78.7, and on LVOS v2 the numbers start from a higher base, 77.8 to 81.4 for Tiny and 83.0 to 85.2 for Large. The SAM 2.1 checkpoints table repeats the pattern with higher baselines across the board, Tiny at 75.1 to 78.9 on SA-V val and Large at 78.6 to 81.1. Both tables are in the README, and both are labelled as one-to-one comparisons between SAM 2 and SAM2Long using the same checkpoint. ## The memory tree holds competing hypotheses instead of committing to one The mechanism is described as a training-free memory tree that reduces the risk of error propagation over time, and it works by maintaining diverse segmentation hypotheses and dynamically pruning less optimal paths as the video progresses. That is the whole idea, and it is worth contrasting with what SAM 2 does on its own. A tracker with a single memory of the object makes a decision at each frame and carries it forward. If that decision is wrong, the wrong state becomes the input to the next decision, and the error compounds. The tree keeps alternatives alive, so a hypothesis that looked plausible in frame 40 can still be present in frame 200 in case the earlier evidence was misleading. The pruning is what stops the tree from growing without bound. As the video progresses, the less optimal paths are removed, so the tree keeps a set of candidates that are still consistent with the evidence rather than every hypothesis ever considered. The README's claim is that this improves segmentation without the need for additional parameters or further training, and that it maximises the potential of SAM 2 in complex video scenarios. Two properties follow from the description. It is training-free, so there is no fine-tuning step and no new weights to train or ship. And it adds no parameters, so the memory cost is in bookkeeping rather than in a larger model. The point about not needing additional training is also why the project's own footprint is small. The repository top level holds img/, a sam2/ directory, sav_dataset/, tools/, setup.py, pyproject.toml, a docker-compose.yaml, a backend.Dockerfile and a .clang-format. The method lives in the sam2/ tree as a modification of the upstream package rather than as a separate model, and the documentation points at three specific places for the rest: tools/README.md for the inference instructions, sav_dataset/README.md for the evaluation code, and the checkpoints directory for the weights. ## Setup means installing SAM 2 first, then downloading the weights The installation section is deliberately thin, and that is a decision rather than an omission. It says to follow the installation instruction of the official SAM 2 repo, and if you encounter issues running the code it is recommended to create a new environment specifically for SAM2Long instead of sharing it with SAM 2, with a link to an issue comment for further details. That separate-environment advice is worth taking seriously. The upstream SAM 2 install includes a compiled CUDA extension, and the setup.py in this repository shows the switch: BUILD_CUDA is read from the SAM2_BUILD_CUDA environment variable and defaults to on, so you can export SAM2_BUILD_CUDA=0 to skip the CUDA build. The required packages list torch 2.3.1 or newer, torchvision, numpy, tqdm, hydra-core, iopath and pillow, and the optional extras include notebooks, an interactive-demo group and a dev group. The package metadata in setup.py is still the upstream one: NAME is SAM-2, VERSION is 1.0, and the license field says Apache 2.0, with a copyright header from Meta Platforms. That is inherited from the SAM 2 codebase it builds on, and it is worth reading alongside the project's own licence, because the two cover different things. Once SAM 2 is installed, the checkpoints come from a script: ```bash cd checkpoints && \ ./download_ckpts.sh && \ cd .. ``` The README says all the model checkpoints can be downloaded by running that. It is a two-step process overall: environment first, weights second, and neither step is specific to SAM2Long beyond the environment isolation advice. For running the thing, the README defers to a file. The inference instruction is in INFERENCE.md, which the link points at tools/README.md, and the evaluation code for the SA-V dataset is in sav_dataset/README.md. For LVOS results on seen and unseen categories, the README points at a separate evaluation repository, LingyiHongfd/lvos-evaluation, which is a signal that the LVOS numbers in the results table were produced with someone else's evaluation code rather than the one in this repository. A live demo is available as a Hugging Face Space, Mar2Ding/SAM2Long-Demo, and the README's news section records it going live on 2024-12-9. ## The demo service wants a GPU, and the compose file reserves one The docker-compose.yaml in the repository is the interactive demo stack rather than a training setup, and it is where the hardware expectation is stated explicitly. ```yaml deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] ``` The backend service reserves one NVIDIA GPU device, and the compose file also sets a long list of environment variables that describe the video handling: FFMPEG_NUM_THREADS, VIDEO_ENCODE_CODEC set to libx264, VIDEO_ENCODE_CRF at 23, VIDEO_ENCODE_FPS at 24, and a maximum width of 1280 with a maximum height of 720. The default video path points at gallery/05_default_juggle.mp4, and the data directory is mounted read-write as ./demo/data/:/data/:rw. The backend runs gunicorn with one worker and two threads, and the comment explains why two threads are the minimum: the inference API needs to have at least 2 threads to handle an incoming parallel cancel propagation request. The frontend service is exposed on port 7262 and the backend on 7263, and API_URL points at http://localhost:7263 so the browser talks to the backend through the host. The separate environment variable list also has SERVER_ENVIRONMENT set to DEV, which tells you this is configured for development rather than for a deployed instance. The overall picture is a research demo: a small service that encodes video with ffmpeg and runs inference on a reserved GPU, with a frontend that talks to it. For evaluation work, the route is the scripts in tools/ and the SA-V evaluation code, not this stack. ## The licence split is the detail to read before you build on it The repository carries a licence that GitHub could not classify, and the README spells out the arrangement in two sentences that matter for any commercial use. The majority of the project is released under the CC-BY-NC 4.0 licence as found in the LICENSE file. The original SAM 2 model checkpoints and SAM 2 training code are licensed under Apache 2.0. So the upstream parts are permissively licensed and the SAM2Long-specific parts are not. That split has a practical consequence. CC-BY-NC is a Creative Commons licence with a non-commercial clause, and it is not the same category of licence as the Apache 2.0 that covers the model weights and the training code. A team shipping a product that uses SAM 2 checkpoints for inference can point at the Apache 2.0 grant for those weights, and a team shipping SAM2Long's own code has to deal with the non-commercial restriction on that code. Whether the method itself, as distinct from the code, is covered by the CC grant is a question for a lawyer rather than for this article, but the boundary is drawn in the README and it is drawn explicitly. There is also a question of whether a training-free inference-time modification is a derivative work of the Apache 2.0 licensed code at all. That is genuinely unsettled in practice for research code that modifies an upstream repository in place, and it is one of the reasons the method's reach is limited. The project is from CUHK and Shanghai AI Lab, with the paper at arXiv 2410.16268 and accepted at ICCV 2025. The last push to the repository was on 2026-08-14, and there are no GitHub releases, so there is no version to pin and no changelog. ## Where SAM2Long is the wrong tool Three limits, and the first is about where the method applies at all. The first is that this fixes long video, not segmentation quality. If your video is short, or if the object stays visible and unoccluded, SAM 2's own tracker is already doing fine and the memory tree is overhead with no payoff. The gains in the tables are on SA-V and LVOS, which are long-term benchmarks with heavy occlusion, and the smallest gains appear where SAM 2 was already strongest, the Large backbone on SA-V test moving from 75.5 to 80.8 against Tiny's 74.6 to 78.7. If you do not have that failure mode, you are paying for a mechanism that never fires. The second is the cost of running it. The method is training-free, which removes the training cost, but the inference cost is not free: maintaining several hypotheses and pruning them means more state to carry through a long video than a single tracked object, and the demo stack reserves a GPU and encodes video with ffmpeg to feed it. For a team that needs to segment thousands of hours of footage on a fixed budget, the arithmetic of how much slower each video takes is the deciding number, and the README does not give a throughput figure. The third is the operational looseness of the repository itself. There are no GitHub releases, so there is no version to pin, and the install route is to follow the upstream SAM 2 instructions with an environment-separation warning attached, which suggests the integration has sharp edges. The setup.py still carries upstream SAM 2 metadata with version 1.0 and an Apache 2.0 licence field, which is a small but real hazard for anyone who reads the package metadata to decide what they are licensing. The evaluation code for LVOS is in a separate repository, so the headline LVOS numbers are not reproducible from this repository alone. The alternative approach for the same problem is to tune SAM 2 at inference time rather than to change its memory. That means adjusting the memory length and the number of previous frames the tracker attends to, or running the base model on sampled keyframes and propagating masks between them with a separate tracker, so a single object hypothesis is replaced by a deliberate keyframe schedule. The difference in approach is that SAM2Long keeps the uncertainty inside the model and resolves it with the evidence, while the keyframe approach discards the frames where the model is unreliable, which is simpler and cheaper but gives up on frames that carry useful information. If your failures are occlusion-driven and the objects reappear, the tree is the better bet. If your failures are resolution-driven or the objects genuinely leave the frame, neither will help much. ## Conclusion Adopt SAM2Long if you are running SAM 2 on video long enough for the tracker to lose an object, because the memory tree keeps competing hypotheses and prunes them as evidence arrives, and the reported gains hold across all four SAM 2 and SAM 2.1 backbone sizes on SA-V and LVOS v2. Do not adopt it for a commercial product without checking the licence, because the majority of the project is CC-BY-NC 4.0 and only the original SAM 2 checkpoints and training code are Apache 2.0. Verify first that your environment matches the official SAM 2 install, that you run inference from the instructions in tools/README.md rather than the SAM 2 defaults, that the checkpoints download script has actually fetched the weights, and that the backend service has a GPU reserved, since the compose file asks for one NVIDIA device for the inference API. ## FAQ ### What is SAM2Long and what problem does it solve? It is the official implementation of an ICCV 2025 paper that addresses SAM 2's error accumulation on long videos, where a tracker that loses an object never recovers. It keeps several segmentation hypotheses in a training-free memory tree and prunes the weaker ones as the video progresses. ### How do I install SAM2Long? Follow the installation instructions of the official SAM 2 repo first, and create a separate environment for SAM2Long rather than sharing one with SAM 2 if you hit problems. Then download the model checkpoints by running the download script in the checkpoints directory. ### How much does SAM2Long improve on SAM 2? The README reports an average improvement of 3 in J and F scores across all 24 head-to-head comparisons on long-term video datasets like SA-V and LVOS. On SA-V validation with SAM 2 checkpoints, Tiny goes from 73.5 to 77.0 and Large from 76.3 to 80.8. ### Does SAM2Long need to be trained or fine-tuned? No. The memory tree is described as training-free and adds no additional parameters, so it improves segmentation without further training. The checkpoints are the ones from SAM 2 and SAM 2.1. ### What licence applies to SAM2Long? The majority of the project is released under CC-BY-NC 4.0 in the LICENSE file, which GitHub could not classify. The original SAM 2 model checkpoints and SAM 2 training code are licensed under Apache 2.0, so the two parts carry different terms. ### Does the SAM2Long demo need a GPU? Yes. The docker-compose.yaml reserves one NVIDIA GPU device for the backend service, with capabilities set to gpu, and the backend runs gunicorn with one worker and two threads because the inference API needs at least two threads to handle a parallel cancel propagation request. ## Sources - [Issues](https://github.com/Mark12Ding/SAM2Long/issues) - [Mark12Ding/SAM2Long on GitHub](https://github.com/Mark12Ding/SAM2Long) - [Project website](https://mark12ding.github.io/project/SAM2Long/) - [README](https://github.com/Mark12Ding/SAM2Long/blob/main/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mark12ding-sam2long