# aot-benchmark is the reference code for a family of video segmentation models, not a library > aot-benchmark is a modular PyTorch implementation of the AOT series of video object segmentation frameworks, covering AOT, DeAOT, AOST and a panoptic variant, with training and inference scripts rather than a packaged API. It is a research code release, and it says so. **yoxu515/aot-benchmark** — An efficient modular implementation of Associating Objects with Transformers for Video Object Segmentation in PyTorch - Repository: https://github.com/yoxu515/aot-benchmark - Stars: 595 · Forks: 101 - Language: Python - License: BSD-3-Clause - Published: 2026-10-22 · Updated: 2026-10-22 · Language: en - Canonical page: https://hysenlabs.com/projects/yoxu515-aot-benchmark ## One repository, four generations of the same idea aot-benchmark is described as an efficient modular implementation of Associating Objects with Transformers for Video Object Segmentation in PyTorch. The README presents it as a modular reference PyTorch implementation of the AOT series frameworks, and the series is where the value is. AOT is Associating Objects with Transformers for Video Object Segmentation, from NeurIPS 2021, with a score of 8/8/7/8. DeAOT is Decoupling Features in Hierarchical Propagation for Video Object Segmentation, from NeurIPS 2022 as a Spotlight. AOST is the extension of AOT, described as a more flexible framework supporting run-time speed-accuracy trade-offs, and the News section records that it was accepted by TPAMI in March 2024. There is also a panoptic branch. The code of PAOT has been released in the paot branch of the same repository, with a separate benchmark called VIPOSeg proposed for panoptic VOS, and the README says PAOT is designed to tackle the challenges in panoptic VOS and achieves SOTA performance. PAOT consists of a multi-scale architecture of LSTT, the same as MS-AOT in VOT2022, plus panoptic ID banks for thing and stuff. So the repository is a family, not a single model. The most recent code, the panoptic variant, lives on a branch rather than on main, which is a detail worth knowing before you clone the default branch and go looking for it. The licence is BSD-3-Clause and the primary language is Python. ## The mechanism is memory, and the interesting change is what gets remembered The AOT idea is to associate objects across frames using transformers, so a mask given on one frame propagates to later frames without a detector. The DeAOT change is described in the README's News entries as decoupling features in hierarchical propagation, and the mechanism that made it win is spelled out in the competition results. The DeAOT-based tracker stores object-wise long-term memories instead of frame-wise long-term memories. That distinction is the whole point, and the README explains the consequence: storing memories per object avoids the memory growth problem when processing long video sequences, and produces better results when handling multiple objects. The reasoning is straightforward once stated. A frame-wise long-term memory has to grow as the video lengthens, because each new frame adds information to a single accumulating record. An object-wise memory stores one record per object, so the memory size depends on the number of objects you track, not on the length of the video. For a long sequence with a handful of objects, that is the difference between a memory that fits and one that does not. The MS-AOT and DMAOT entries describe the multi-scale versions of the same idea, with MS-AOT winning the two real-time segmentation tracks at VOT 2022 and DMAOT ranked first at VOTS 2023. The README also records that a DeAOT-based tracker ranked first in two tracks of the EPIC-Kitchens challenges in June 2023, and that SAM-Track, which merges SAM with DeAOT, was released in April 2023 as a separate project. AOST then adds the run-time speed-accuracy trade-off, described as the first scalable VOS framework supporting them, from real-time efficiency to SOTA performance. ## The numbers, as the project states them The Highlights section of the README lists specific figures, and they are worth reading with the attribution that they are the project's own claims under stated conditions. On performance, up to 85.5 percent with the R50-AOTL variant on YouTube-VOS 2018, and 82.1 percent with the SwinB-AOTL variant on DAVIS-2017 Test-dev, under standard settings. The README is careful to add what standard settings excludes: no test-time augmentation and no post processing. On efficiency, up to 51fps with the AOTT variant on DAVIS-2017 at 480p even with 10 objects, and 41fps on YouTube-VOS at 1.3x480p. The claim that follows is the interesting one: AOT can process multiple objects, less than a pre-defined number with 10 as the default, as efficiently as processing a single object. The README also says the project supports inferring any number of objects together within a video by automatic separation and aggregation, which is how it gets past the 10 object default. The other highlights are engineering features rather than numbers: multi-GPU training and inference, mixed precision training and inference, and test-time augmentation with multi-scale and flipping. All of these numbers live in the repository as reproducible artefacts, because the README points to MODEL_ZOO.md for pre-trained models, benchmark scores and pre-computed results reproduced by the project, and the Highlights section links specific entries in that file for the YouTube-VOS 2018 validation set and the DAVIS-2017 test set. If you want to check a claim, the checkpoint and the score are both named rather than asserted. ## Requirements, and one extension you have to compile yourself The Requirements section lists Python3, pytorch 1.7.0 or newer together with torchvision, opencv-python and Pillow. Then there is the one that decides whether your setup works on day one: Pytorch Correlation, with a recommendation to install it from source rather than from a package index. ```bash git clone https://github.com/ClementPinard/Pytorch-Correlation-extension.git cd Pytorch-Correlation-extension pip install . cd .. ``` That is a CUDA extension built and installed into your environment from a third party repository. It is the mechanism behind the correlation based matching that the transformer association depends on, and there is no prebuilt path offered in the README. So the setup order is: get a compatible PyTorch and CUDA, compile the correlation extension, then run the project. If your GPU is newer than the extension was last built against, you will be reading compiler errors rather than model code, and that is the first thing to budget time for. One optional dependency is listed: scikit-image, needed if you want to run the demo. The demo itself is a panoptic propagation demo that the README says will propagate more than 40 objects, including semantic regions, which is a good demonstration of the automatic separation and aggregation behaviour described in the highlights. ## Training and inference are shell scripts, not an API The repository layout explains the shape of the project. There are configs/, dataloaders/, datasets/, networks/, tools/ and utils/ directories, plus source/, pretrain_models/, MODEL_ZOO.md, and the two entry points at the top level, train_eval.sh and inference.sh. That layout is the standard one for a research code release. The model definitions live in networks/, the data pipeline is split across dataloaders/ and datasets/, the configuration is in configs/, and the training and evaluation entry points are shell scripts that drive the whole thing. The consequence for a user is direct. There is no pip install and no importable package here. You clone the repository, put your configuration in place, and run the scripts. Reproducing a result means following the recipe for that entry in MODEL_ZOO.md rather than calling a function. The pretrain_models/ directory at the top level holds the checkpoints, which is where the model zoo actually lives on disk, and the README points at MODEL_ZOO.md for the mapping from checkpoint to dataset and score. This is a deliberate shape for a paper repository. It optimises for the ability to modify the method and re-run the experiments, not for the ability to call a segmentation model from your own application. ## What you give up, and what a production use would need There are real constraints, and they come from the design rather than from bugs. First, there is no package. No PyPI release, no versioned distribution, and the repository has no GitHub releases at all. The last push was on 2026-04-07, so the code is being worked on, but there is no tag to pin and no changelog for consumers. A dependency on it is a dependency on a commit hash. Second, the environment is demanding. A custom CUDA extension built from source, a specific PyTorch floor, and a dataset layout in the standard research format. None of that is unreasonable for a research project, and all of it is unreasonable for a service that has to run on a schedule. Third, the scope is a benchmark. The demos and the model zoo are for video object segmentation on DAVIS and YouTube-VOS style data with a mask given on the first frame. There is no inference service, no model server, and no documented API for calling a trained model on your own video. The alternative worth naming is SAM-Track, which the README points to as a separate project from the same author, released in April 2023. Its difference in approach is that it merges SAM, the segmentation model, with DeAOT, so that segmentation comes from a promptable segmentation model rather than from a mask given on the first frame, and the tracking of any objects in videos comes from the DeAOT propagation. If your problem is that you do not want to hand annotate the first frame, that is a different tool with a different dependency profile, and the README makes the comparison explicit rather than leaving you to guess. ## Maintenance, licensing, and how to judge a research repository The repository is not archived and the last push was on 2026-04-07. There are no GitHub releases, and the Recent releases list is empty for the same reason. The licence is BSD-3-Clause, with the text in the LICENSE file, which is permissive and has no copyleft obligation on anything that uses or modifies the code. The most recent substantive news entry in the README is from March 2024, when AOST was accepted by TPAMI. The competition results that are documented go back to VOT 2022 in October 2022, and the panoptic branch and VIPOSeg benchmark date from July 2023. So the headline achievements in the README are two to four years old, while the code has continued to receive pushes. How to read that combination is a judgement call, and it depends on what you want. For reproduction, what matters is that the checkpoints and scores are published and named, which they are, in MODEL_ZOO.md. For a production dependency, what matters is that there is no versioned release and no compatibility promise, and there is not. The project also has an academic framing that is worth respecting. It is a reference implementation of published research, with the badges linking to Papers With Code entries for the SOTA results on DAVIS and YouTube-VOS. If you use it in a paper, the citation is the paper, and the code is the artefact that goes with it. ## Conclusion Adopt aot-benchmark if you are doing research on video object segmentation or want to reproduce the numbers behind the AOT, DeAOT and AOST papers, and you are prepared to work with training scripts rather than an installable library. Do not adopt it as a production dependency for a segmentation service, because it publishes no releases, no package on PyPI and no stability promise. Before you build on it, read MODEL_ZOO.md for the checkpoints and reproduced scores, and confirm you can install the Pytorch-Correlation extension from source, because that extension is a hard requirement rather than an optional extra. ## FAQ ### What is aot-benchmark? It is a modular reference PyTorch implementation of the AOT series frameworks for video object segmentation, covering AOT from NeurIPS 2021, DeAOT from NeurIPS 2022, the AOST extension, and a panoptic variant called PAOT released on the paot branch. ### What is the difference between AOT and DeAOT? DeAOT decouples features in hierarchical propagation, and the README explains that it stores object-wise long-term memories instead of frame-wise long-term memories. That avoids the memory growth problem on long video sequences and produces better results with multiple objects. ### What do I need to install to run aot-benchmark? Python3, pytorch 1.7.0 or newer with torchvision, opencv-python and Pillow, plus the Pytorch-Correlation extension, which the README recommends installing from source. scikit-image is optional and needed for the demo. ### Where are the pre-trained models and the benchmark scores? The README points to MODEL_ZOO.md for pre-trained models, benchmark scores and pre-computed results reproduced by the project, and the repository has a pretrain_models/ directory. The Highlights section links specific entries for the YouTube-VOS 2018 validation set and the DAVIS-2017 test set. ### How do I run the demo for aot-benchmark? Install the optional scikit-image dependency, which the README says is needed to run the demo. The demo demonstrates panoptic propagation and will propagate more than 40 objects, including semantic regions. ## Sources - [Issues](https://github.com/yoxu515/aot-benchmark/issues) - [License: BSD-3-Clause](https://github.com/yoxu515/aot-benchmark/blob/main/LICENSE) - [README](https://github.com/yoxu515/aot-benchmark/blob/main/README.md) - [yoxu515/aot-benchmark on GitHub](https://github.com/yoxu515/aot-benchmark) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/yoxu515-aot-benchmark