Segment and Track Anything: model tracking and reproducible builds
An open-source project dedicated to tracking and segmenting any objects in videos, either automatically or interactively. The primary algorithms utilized include the Segment Anything Model (SAM) for key-frame segmentation and Associating Objects with Transformers (AOT) for efficient tracking and propagation purposes.
At a glance
- What is it?
- SAM-Track pairs Segment Anything for key-frame segmentation with DeAOT for propagation, and its Dockerfile pins twenty six Python dependencies to exact versions before cloning a third repository from its main branch. The newest release is 1.6 from 2024, the 2.x interface is still an unchecked box, and the adoption table has one row.
- Who is it for?
- This is a research pipeline with a research pedigree rather than a library with a support commitment, and the two facts to weigh are the dependency graph and the release gap. The Dockerfile is unusually disciplined about Python versions, which is worth copying, but it clones GroundingDINO from main during the build, so two builds a month apart can differ without any change in this repository.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 135 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- z-x-yang/Segment-and-Track-Anything3,148 ★ · +9
| Repository | 2026-09-25 | 2026-11-15 |
|---|---|---|
| z-x-yang/Segment-and-Track-Anything | 3.1k | 3.1k |
9 Stars in the last 51 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
SAM segments key frames and DeAOT tracks objects
The pipeline is two models and a division of labour. SAM, the Segment Anything Models, handles the key frames, both automatically and interactively, and it is what finds objects that have just appeared. DeAOT, Decoupling features in Associating Objects with Transformers, published at NeurIPS 2022, handles tracking and propagation of everything already identified. So the segmentation model never follows anything and the tracking model never decides what an object is. The naming drifts a little in the repository: the project blurb calls the tracker Associating Objects with Transformers, the README calls it DeAOT and links the aot-benchmark repository, and the code lives in an `aot` directory with a module called aot_tracker.py at the root. Three names for one component is a small thing, but it is the kind of thing that makes a search for the wrong term fail.
Long term memory is the argument that decides whether long videos work
The only two tuning arguments the project calls out by name were added for AOT-L in April 2023, and the README explains both in more detail than any other option. `long_term_memory_gap` controls how often the model adds a new reference frame to its long term memory, and during mask propagation the model matches the current frame against the reference frames stored there. A properly chosen gap value is what buys performance. The second argument, `max_len_long_term`, exists for a different reason: to avoid memory explosion on long videos, once the stored frame count reaches that value the oldest frame is discarded and a new one added. That is a bounded cache with a write policy, and it is the parameter to reach for first when a long video runs out of memory.
The newest release added audio, and the 2.x interface is still a checkbox
The roadmap at the bottom of the README is a checklist, and its shape is the clearest statement of where the project is. Three items are ticked: the Colab notebook, completed on April 25th 2023, the 1.0 interactive WebUI on April 23rd, and the 1.5 WebUI on April 26th. The 2.x version is not ticked. Its promises are written out, though: the segmentation model would offer two options, SAM and SEEM, a fixed category object detection result could be shown as a prompt, and objects could be added and modified during a video. What shipped instead was version 1.6 on 2024-04-25, whose new feature was audio grounding, tracking the sound-making object in a video's soundtrack. The most recent commit on main is dated 2026-07-03, so the code has moved since that release without a new tag.
Three tutorials for three input styles, each mirrored on two platforms
The tutorial links are named after the interaction they teach, which is the clearest way to understand the feature history. The 1.0 tutorial is labelled click and brush, matching a WebUI built on strokes and clicks. The 1.5 tutorial is labelled text, matching the release that added text prompts and the ability to add several tracked objects interactively. The 1.6 tutorial carries audio in its name. Each of the three 1.0 videos also exists twice, once on YouTube and once on Bilibili, covering the same three subjects: modifying a single object mask for the first frame, adding an object by click, and adding an object by stroke. The 1.5 and later demos follow the same two platform pattern. The 1.0 version is described as a developer version, with an invitation to report bugs to the authors directly.
The Dockerfile clones a dependency from its main branch
The container recipe is disciplined about Python and careless about one Git dependency. It starts from a pinned PyTorch image, pytorch/pytorch:2.0.1-cuda11.8-cudnn8-devel, sets CUDA_HOME and DEBIAN_FRONTEND, installs a short list of system packages including ffmpeg, cmake and python3-dev, copies the whole tree into /workspace, and then pins many Python dependencies to exact versions; the same recipe also leaves wget and gdown unpinned. After that comes the line that undoes the reproducibility:
git clone -b main https://github.com/IDEA-Research/GroundingDINO.git && \
cd GroundingDINO && \
pip install -e . --no-build-isolationA branch reference at build time means the image is not a function of this repository's commit. The entry point is `python app.py`, and a GroundingDINO notebook is the reason the clone is there at all.
Twenty six exact pins, and a formatter among them
Read the pip list as a snapshot of one moment in the ecosystem. transformers is at 4.30.2, numpy at 1.26.4, opencv-python at 4.10.0.84, scikit-image at 0.24.0, matplotlib at 3.9.2, supervision at 0.22.0, pycocotools at 2.0.8. The web layer is gradio 3.39.0 with gradio_client 0.5.0, and behind it fastapi 0.100.1 and starlette 0.27.0, which pair with pydantic 1.10.13 from the first generation of that library. Two entries do not belong in a runtime list at all: yapf 0.40.2 is a source formatter, and wget is installed a second time through pip after already coming from apt. Only two names are unpinned, wget and gdown, and gdown is the one that fetches model weights, so the download path is the least reproducible part of the image.
The adoption table is one row wide
The section titled Impact makes a strong claim in one sentence, that this project has been adopted as core infrastructure in published robotics research from MIT CSAIL and Harvard SEAS. Underneath it is a four column table with a single row. The project is Follow Anything, abbreviated FAn, the institutions are MIT CSAIL, Harvard SEAS and Project CETI, the venue is IEEE RA-L, and the description is an open-vocabulary real-time drone tracking system. So the entire adoption claim in the repository is one paper, credited to three groups, in one venue. That is a real citation and the README links it to the publisher, but a table with a single row reads as a placeholder for a section that was meant to grow.
Two notebooks in a repository labelled as notebooks
The primary language recorded for this repository is Jupyter Notebook, and there are exactly two notebooks at the root, demo.ipynb and demo_instseg.ipynb, the second being the Grounding-DINO detection example added in May 2023. Everything else in the tree is Python scripts at the top level: app.py, which is what the container runs, seg_track_anything.py, SegTracker.py, aot_tracker.py, img2vid.py, prepare.py and model_args.py, plus directories for the vendored sam code, the aot code, tool, script, tutorial, assets and demo material. Two licence files sit side by side as well, LICENSE.txt at the root and licenses.md beside it, which is the usual arrangement when third party components with their own terms are bundled in a directory.
Editorial conclusion
This is a research pipeline with a research pedigree rather than a library with a support commitment, and the two facts to weigh are the dependency graph and the release gap. The Dockerfile is unusually disciplined about Python versions, which is worth copying, but it clones GroundingDINO from main during the build, so two builds a month apart can differ without any change in this repository. The AOT long term memory arguments are the part of the interface that decides whether a long video is feasible at all, so set them deliberately rather than by default. And if you need a citation for adoption, the README offers exactly one: a drone tracking paper from MIT CSAIL, Harvard SEAS and Project CETI in IEEE RA-L.
Frequently asked questions
What does SAM-Track use for segmentation and what does it use for tracking?
SAM, the Segment Anything Models repository, does the automatic and interactive key-frame segmentation and finds newly appearing objects, while DeAOT, Decoupling features in Associating Objects with Transformers from NeurIPS 2022, tracks and propagates every object already identified.
How do I run Segment and Track Anything?
The repository ships a Dockerfile whose command is python app.py, and an online demo runs in Colab. No package index coordinate appears in the README, so the container or the notebook is the documented path.
What does long_term_memory_gap do in Segment and Track Anything?
It controls how often the AOT model adds a new reference frame to its long term memory, which it matches the current frame against during mask propagation. The companion argument max_len_long_term caps the stored frames so long videos do not exhaust memory, discarding the oldest frame when the cap is reached.
Does Segment and Track Anything support text prompts?
Yes, from the 1.5 version of the interactive WebUI. The 1.0 version worked with strokes and clicks, and 1.5 added text prompts along with the ability to add several tracked objects interactively.
Which license does Segment and Track Anything use?
The Affero General Public License version 3. The packaging data gives AGPL-3.0, LICENSE.txt sits at the root, and a separate licenses.md file sits beside it for the bundled third party components.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/z-x-yang-segment-and-track-anything)Star history chart for your README
[](https://hysenlabs.com/projects/z-x-yang-segment-and-track-anything)Was this useful?