SysCV/cascade-detr: the DETR variant that scores queries by expected IoU

Attention is limited to the previous box prediction

The first contribution is structural and is described in one sentence: the Cascade Attention layer explicitly integrates object-centric information into the detection decoder by limiting the attention to the previous box prediction. The point is that a decoder stage should refine a region rather than re-scan the whole image, so each stage's attention is focused by what the stage before it predicted. That is where the name comes from. The cascade here is a sequence of refinements of one prediction, not a cascade of feature extractors, and the authors report that this joint treatment of domain generalisation and localisation accuracy is what produces their gains: improvements of over 10 mAP on some of the benchmark datasets, with the gap widening under stringent quality requirements.

Queries are scored by expected IoU, not classification

The second contribution is about how confident the model is allowed to sound. Instead of relying on classification scores, the model predicts the expected IoU of the query, and the README argues this leads to substantially more well-calibrated confidences. The distinction matters in practice because a classification score and a localisation quality are different quantities: a box can be confidently the right class and badly placed. Scoring by expected overlap collapses the two into one number that says something about the box itself, which is what makes a threshold meaningful across datasets with different characteristics. Both changes are described as jointly tackling generalisation and accuracy rather than as independent additions.

The reference stack is Python 3.8 and CUDA 11.0

The installation section names one combination and immediately hedges it, stating that the project uses Python 3.8 with PyTorch 1.7.1 and CUDA 11.0, and that other versions may also be available. The README also insists that both PyTorch and TorchVision must be installed with CUDA support. That hedge matters because the requirements file is looser than the example: the dependency floor asked for is torch 1.5.0 or newer and torchvision 0.6.0 or newer, well below the pinned versions in the conda recipe, so nothing in the dependency list will stop you from trying a newer toolkit. Two entries install straight from GitHub rather than from a package index, with a subdirectory and an egg name in the URL: pycocotools and panopticapi. The rest of the list is conventional, with cython needed for the compiled operator below, submitit apparently for job submission, timm for backbones and yapf for formatting.

The deformable attention operator is compiled and tested

The last step of setup is a build, and it has its own verification.

# 4. deformable attention
cd cascade_dn_detr/models/cascade_dn_detr/ops
sh make.sh
# unit test (should see all checking is True)
python test.py
cd ../../..

A shell script compiles the operator and a Python test checks it, with the expected signal being that every check reports True. The path tells you where the model lives inside the repository, four levels down from the top, and the trailing `cd` returns you to the root so the next instruction starts from a known place. If this step fails on a current machine, the cause is almost certainly the compiler toolchain rather than the detector, since it depends on cython and on the CUDA headers being present.

Five of the ten benchmark datasets come from Roboflow

UDB10 collects ten datasets across domains, all fetched from public links, and the list is worth reading because the formats differ. COCO contributes the standard 2017 train and validation split. UVO is the unidentified video object dataset at version 0.5 with its default split. Cityscapes comes from its own site, but its detection annotations in COCO format are obtained from another project. The remaining six are hosted elsewhere: brain tumor, Document Parts, Smoke, Egohands, PlantDoc and People in Painting are all Roboflow universe downloads, and for each the README recommends taking the COCO JSON format rather than the default. BDD100K is the exception that needs work: it comes from its official site and the annotations have to be converted, though pre-converted COCO-format files are provided if you would rather skip that step.

Everything real lives in one subtree

The top level of the repository holds four entries: the license, this README, the requirements file, and a directory named cascade_dn_detr. That single directory contains the working code, and the README treats entering it as the next step.

cd cascade_dn_detr

Once inside, there are detailed training and evaluation scripts along with pretrained checkpoints, and a README of its own which is where the expected dataset directory structure for training and evaluation is documented. That structure is not in the top-level file, which is worth knowing before you start downloading anything, since the layout the training script expects is defined one level down. Naming is slightly awkward to keep straight: the repository is cascade-detr and the package inside it is cascade_dn_detr, which is also the name of the model directory holding the compiled operator.

Seven authors, and a request to cite the baseline numbers

The citation section asks two things of you: to star the repository if the work is useful, and to cite it if you refer to the baseline results the authors provide. The entry names seven authors, from Mingqiao Ye and Lei Ke through Siyuan Li, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan and Fisher Yu, with the venue given as ICCV and the year 2023. The ask about baseline results is the detail with weight: those per-dataset training results are what a comparison paper would lean on, so the request to cite them is really a request to attribute the numbers rather than the code alone. The repository is under the Apache-2.0 license, was last pushed on 2023-09-12, and publishes no releases, so the checkpoints in the repository are the artefact to plan around.

Editorial conclusion

Use this if you are working on DETR-family detection and care about localisation accuracy and confidence calibration rather than only average precision, since both contributions target that. Two things to check before you start. The reference environment is a 2020-era stack, with Python 3.8, PyTorch 1.7.1 and CUDA 11.0, so confirm your driver and toolkit before committing. And the deformable attention operator is compiled during setup and unit tested, so a toolchain failure there is a build environment problem rather than a model problem. The last push to main is dated 2023-09-12 and there are no releases.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository