RT-DETRv4 distils vision foundation models into a real-time detector

What the paper adds to the RT-DETR family

RT-DETRv4 is the new version of the state-of-the-art real-time object detector family, RT-DETR, and its central contribution is described as a cost-effective and adaptable distillation framework that uses the representations of Vision Foundation Models to improve lightweight detectors.

Read that carefully, because it defines the scope. This is not a new detector architecture competing with the foundation models. It is a way of distilling what a large vision foundation model knows into a small detector that runs at real-time speed. The detector is the artefact; the foundation model is the teacher.

That framing explains both the name and the structure. Painlessly Furthering is the claim, and the paper is titled RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models. The arXiv identifier in the badge is 2510.25257, and the repository is the official implementation of that paper.

The paper has been accepted to ECCV 2026, announced on 2026-06-18, and the repository history is short and dated. The repo was created on 2025-10-30 with a note that code would be open-sourced soon, the code, configs and checkpoints were fully released on 2025-11-17, and the ECCV acceptance came on 2026-06-18. The last push was on 2026-07-06, the repository is not archived, the licence is Apache-2.0 and the primary language is Python.

The four checkpoints and what they cost

The performance table is the most useful single artefact in the README, and it gives accuracy and speed together, which is the trade-off that matters for a real-time detector.

| Model | AP | AP50 | AP75 | Latency (T4) | FPS (T4) | | :--- | :---: | :---: | :---: | :---: | :---: | | RT-DETRv4-S | 49.8 | 67.1 | 54.0 | 3.66 ms | 273 | | RT-DETRv4-M | 53.7 | 71.0 | 58.4 | 5.91 ms | 169 | | RT-DETRv4-L | 55.4 | 73.0 | 60.3 | 8.07 ms | 124 | | RT-DETRv4-X | 57.0 | 74.6 | 62.1 | 12.90 ms | 78 |

The README states that RT-DETRv4 achieves new state-of-the-art results on the COCO dataset, outperforming previous real-time detectors, and there is a figure in the repository for accuracy against latency.

The four sizes are backed by HGNetv2 backbones, based on the config file names, which go from rtv4_hgnetv2_s_coco.yml through m and l to x. Each row also links to its own YAML config, a training log and a checkpoint on Google Drive, which is worth noting as a practice: the logs let you inspect how a published number was reached rather than taking it on trust.

The latency and FPS figures are on a T4, and the README is clear about that in the column headers. A T4 is an older datacenter card, so these numbers are the right way to compare the four models against each other, since the hardware is held constant, and they are not the numbers you would see on a modern edge device. That distinction matters for anyone sizing a deployment, and the README does not claim otherwise.

The demo section is a qualitative counterpoint to the table, using the high-speed scene from Furious 7. The claim is that under extreme motion blur, rapid camera pans and chaotic lighting the model maintains consistent tracking and high recall, proving its reliability in dynamic environments.

Setting up the environment and the dataset

The setup is a conda environment with a pinned Python version and a requirements install:

conda create -n rtv4 python=3.11.9
conda activate rtv4
pip install -r requirements.txt

The requirements file is short and tells you a good deal about the implementation:

torch
torchvision

The full list is torch, torchvision, faster-coco-eval at 1.6.5 or newer, PyYAML, tensorboard, scipy, calflops and transformers. That last entry is the one to notice, because transformers is what provides the vision foundation model that the distillation framework draws on. Its presence in the requirements is direct evidence that the teacher model is part of the pipeline rather than a one-off research artefact.

Data preparation for COCO2017 is two steps: download the dataset from OpenDataLab or from the COCO site, then modify the paths in configs/dataset/coco_detection.yml:

train_dataloader:
    img_folder: /data/COCO2017/train2017/
    ann_file: /data/COCO2017/annotations/instances_train2017.json
val_dataloader:
    img_folder: /data/COCO2017/val2017/
    ann_file: /data/COCO2017/annotations/instances_val2017.json

So the configuration is a YAML file per dataset, and the data paths are the only thing that changes for a standard COCO run.

Training on your own data in COCO format

The custom dataset section is the part that decides whether this is usable for a project that is not training on COCO, and it is a three-step procedure.

The first step is to set remap_mscoco_category to False in the config, which prevents the automatic remapping of category IDs to match the MSCOCO categories:

remap_mscoco_category: False

The second is to organise the images into the COCO directory layout, with an images/ directory containing train/ and val/ subdirectories, and an annotations/ directory containing instances_train.json and instances_val.json.

The third is to convert the annotations if they are not already in COCO format. The README supplies a Python stub as a reference rather than a finished converter:

import json

def convert_to_coco(input_annotations, output_annotations):
    # Implement conversion logic here
    pass

if __name__ == "__main__":
    convert_to_coco('path/to/your_annotations.json', 'dataset/annotations/instances_train.json')

The honest reading of that stub is that the conversion is your responsibility. The README says you can use the script as a reference or utilize existing tools, so the library does not ship a converter for arbitrary annotation formats.

The final step is to edit configs/dataset/custom_detection.yml, and the template shows the shape of that file, including the task type, the evaluator configuration with CocoEvaluator and iou_types of bbox, the num_classes field, the remap flag, and the train and val dataloader sections with their transforms, shuffle, num_workers of 4, drop_last, and the BatchImageCollateFunction.

One repository for several detectors

A line in the README that is easy to skim past is worth its own attention, because it changes what the repository is for.

This repository also supports the reproduction of DEIM, D-FINE and RT-DETRv2. Simply run the corresponding configuration files.

Those are three other detectors in the same family of real-time and end-to-end detection work, and the configs are laid out in the tree as configs/deim/, configs/dfine/ and configs/rtv2/. So the repository is a shared training and evaluation codebase with RT-DETRv4 as its headline, and the earlier and sibling models are reproduced by pointing the same engine at a different YAML.

That is a meaningful detail for an evaluator, because it tells you the training infrastructure is general rather than v4 specific. If your question is which detector to use, the answer may come from comparing them under one harness rather than from reading four papers. If your question is how to train, the answer is that the same train.py and the same tools directory handle all of them.

The repository layout supports that reading. The top level has configs/, engine/ for the training and evaluation engine, tools/ for the command line tools, train.py as the entry point, logs/ for the training logs, requirements.txt, figures/ for the README graphics, and a .gitignore. The presence of logs/ in the repository, with a per-model log linked from the performance table, means the training runs are reproducible artefacts rather than numbers quoted from a paper.

Where RT-DETRv4 fits, and where it stops

The comparison worth making is with the other real-time detectors and with the general object detection stacks.

Against the previous RT-DETR versions, the difference is the distillation from a vision foundation model. If that transfers the way the paper claims, the accuracy gain comes without the latency cost of running the foundation model, and the four size variants let you choose where on that curve to sit. Against a general purpose detector such as a YOLO variant, the practical difference is the same: RT-DETRv4 is an end-to-end transformer detector with an NMS-free design, which is why the latency numbers are flat and predictable as the model grows.

The limits are worth being clear about. This is a bounding box detector, and the framework as described does not cover segmentation, pose or oriented boxes, so a project needing those is looking elsewhere. The distillation framework needs the foundation model available, which is why transformers is a dependency rather than an optional extra, and that is a heavier installation than a plain detector.

There is also a reproducibility question that the repository handles unusually well. Each published checkpoint is paired with the exact config file and the training log, so the claim can be audited. The data preparation, on the other hand, is partly on you: the dataset must be in COCO format, and the annotation converter in the README is a stub rather than a tool.

One more caution is the version state. The repository has no GitHub releases, and the news entries show the code, configs and checkpoints only fully released on 2025-11-17. The checkpoints are on Google Drive rather than on a model hub, so if you need a stable artefact with a checksum, download the file and keep the hash yourself.

For a team that needs a fast, accurate detector and can train it on its own data, the four checkpoints and the config-driven training make this a practical starting point, and the shared harness for DEIM, D-FINE and RT-DETRv2 makes it easy to compare.

Editorial conclusion

RT-DETRv4 suits a team that needs a fast detector with better accuracy than the previous real-time family offers and wants to train on its own data in COCO format, since the repo includes the full training pipeline, the dataset configs and the checkpoints. It is the wrong tool if you need a detector for a task other than bounding boxes, and it is the wrong tool if you cannot run the training yourself, because the distillation framework needs the foundation model rather than just the released weights. Before adopting it, download the checkpoint that matches your latency budget from the table in the README, and read the news entries, because the code, configs and checkpoints were only fully released on 2025-11-17 after the repository was created the month before.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. RT-DETRs/RT-DETRv4 on GitHub