# RT-DETR (lyuwenyu/RT-DETR): the official PyTorch and Paddle implementation of a real-time detection transformer

> RT-DETR is the official code release for the CVPR 2024 paper that argues DETRs can beat YOLOs at real-time detection. It ships four separate code trees, two pretrained-weight families and no pip package, which shapes both who should adopt it and how much work that takes.

**lyuwenyu/RT-DETR** — [CVPR 2024] Official RT-DETR (RTDETR paddle pytorch), Real-Time DEtection TRansformer, DETRs Beat YOLOs on Real-time Object Detection. 🔥 🔥 🔥 

- Repository: https://github.com/lyuwenyu/RT-DETR
- Stars: 5,563 · Forks: 664
- Language: Python
- License: Apache-2.0
- Published: 2026-09-22 · Updated: 2026-09-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/lyuwenyu-rt-detr

## What RT-DETR actually is, and who the repository is written for

RT-DETR is the official implementation of two papers: "DETRs Beat YOLOs on Real-time Object Detection" (arXiv 2304.08069, accepted to CVPR 2024) and the follow-up "RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer" (arXiv 2407.17140). The claim in the title is a comparison, and the README backs it with a table of COCO AP, parameter counts, FLOPs and TensorRT FP16 frame rates on a T4. RT-DETR-R18 is listed at 46.5 AP with 20M parameters and 217 FPS; RT-DETR-R101 at 54.3 AP with 76M parameters and 74 FPS. The RT-DETRv2 entries add small increments over their RT-DETR counterparts, for example RT-DETRv2-S at 48.1 AP versus RT-DETR-R18 at 46.5.

The audience is narrower than the paper title suggests. This is a research code release, not a library. There is no pip package, no versioned release, and the repository root is a set of parallel directories rather than a single installable module. If you are evaluating detectors for a production pipeline and you want `pip install` followed by a stable `predict()` call, this is the wrong shape of project. If you are reproducing paper numbers, fine-tuning on your own dataset, or exporting weights to an inference runtime yourself, it is the right one.

## Four code trees, two frameworks, and why that matters before you clone

The top-level layout is the single most important fact about this repository. It contains rtdetr_paddle/, rtdetr_pytorch/, rtdetrv2_paddle/ and rtdetrv2_pytorch/, plus benchmark/, hubconf.py, LICENSE and the two README files. That is four implementations of two model generations across two frameworks. The README's Implementations section maps each one to "code&weight", so the weights and the code are expected to be used as a pair.

This is a deliberate choice, and it has a cost. A bug fix or a new feature does not automatically land in all four trees, and the update log shows entries that name a specific one, such as the RegNet and DLA34 backbones added under rtdetr_pytorch and the RT-DETRv2-S performance improvement noted under rtdetrv2_pytorch. Before you write a single line of training code, decide whether you want the v1 or v2 generation and whether you want PyTorch or Paddle, then work only inside that directory. Reading across trees to assemble a working setup is the most likely way to waste an afternoon.

The benchmark/ directory and hubconf.py are the two pieces that sit outside the framework split. hubconf.py exists so the models can be loaded through torch hub, which the 2024.08.27 update entry confirms. The README does not document what benchmark/ contains.

## Installing RT-DETR from source and running a first detection

There is no package to install. The README points at directories, so the install step is cloning the repository and working inside the tree that matches your framework and model generation. The commands below reflect that layout; the README does not list exact dependency versions, so treat the environment setup as yours to pin.

```bash
git clone https://github.com/lyuwenyu/RT-DETR.git
cd RT-DETR/rtdetrv2_pytorch
```

From there, the tree follows a conventional detector training layout with configs, tools and a model definition. The README's updates mention a profiling utility added for parameter and FLOP statistics, at rtdetrv2_pytorch/tools/run_profile.py, which is the cheapest way to confirm your environment can import the model before you commit to a training run.

```bash
python tools/run_profile.py
```

For inference, the README notes that RTDETR and RTDETRv2 are available in Hugging Face Transformers (referenced through issues #413 and #549) and that RTDETR is available in ultralytics/ultralytics. If your goal is to run a detection on an image today rather than to train, those two routes avoid this repository's source layout entirely, and the README itself points you there. Use the code in this repository when you need the training recipe, the configs, or the export path.

Deployment is the part the README handles least directly. It links a discussion for deployments (issue #95) that names ONNX Runtime, TensorRT and openVINO as supported targets. There is no export command in the README text, so budget time to read that thread and the export scripts inside your chosen tree before promising a deployment date.

## Where RT-DETR is the wrong tool

The clearest limitation is packaging. Nothing in the README describes a supported Python API, a semantic version, or a release artifact. The "Recent releases" field is empty. If your team's dependency policy requires pinned versions from an index, this project cannot satisfy it, and you will be vendoring a git checkout instead.

Second, the accuracy gains in RT-DETRv2 are small and uneven. The README's own table shows +1.6 AP for RT-DETRv2-S over RT-DETR-R18, +1.0 for RT-DETRv2-M (the 31M-parameter variant), +0.6 for the 36M RT-DETRv2-M, +0.3 for RT-DETRv2-L and +0.1 for RT-DETRv2-X. The larger the model, the less the v2 recipe buys you. If you are already running RT-DETR-X, migrating to RT-DETRv2-X for a tenth of a point of AP is not obviously worth the revalidation.

Third, the real-time numbers are TensorRT FP16 figures on a T4. They are not a promise about your hardware, your batch size or your preprocessing. The README gives no latency table for CPU, for ONNX Runtime, or for the mobile path, and the only non-server figure in the README is a community Android port listed at roughly 615 ms per frame on a Pixel 8a for a still image. Anyone planning an edge deployment should treat that as the realistic order of magnitude until they measure their own target.

Finally, the README does not document rollback, checkpoint compatibility between trees, or how to move weights from the Paddle implementations to the PyTorch ones. The 2023.09.19 update mentions uploading PyTorch weights converted from the Paddle version, which tells you conversion happened, not that a conversion tool ships here.

## RT-DETR versus RF-DETR and the Ultralytics route

Two comparisons are worth separating, because they answer different questions.

RF-DETR is a different detector with a similar name, and the search data shows people conflating them. The README covers RT-DETR only; it says nothing about RF-DETR's architecture, training data or licence, so any claim about which is more accurate would be invented. What can be said is structural: RT-DETR's published numbers come from this repository's own COCO table, and comparing them to RF-DETR requires reading RF-DETR's own results under matching input resolution and evaluation protocol.

The more useful comparison is with the Ultralytics route, because it is the one the README endorses. RTDETR is available in ultralytics/ultralytics, which means you can use an RT-DETR model through a package that does have an install story and a documented inference API. The trade-off is control. Going through Ultralytics gives you a maintained wrapper and someone else's config conventions; going through rtdetrv2_pytorch gives you the paper's training recipe, the configs the authors used, and the ability to change the loss, the data augmentation or the backbone without negotiating with an upstream project. Neither is strictly better. If you are shipping an application, the wrapper is usually the right call. If you are ablating the model or fine-tuning on a custom dataset, the source tree is.

## Maintenance, licence and the cost of tracking upstream

The repository is not archived and the last push was on 2026-09-07. The update log runs from April 2023 to November 2025, when the README announces RT-DETRv4 as a separate project in a different GitHub organisation (RT-DETRs/RT-DETRv4). That is a signal about where new work is going: the newest member of the family lives in its own repository, not here. Plan for the possibility that this repository becomes the reference implementation for v1 and v2 while v4 and later generations are developed elsewhere.

Upgrade cost is dominated by the four-tree layout. Because there is no release process, "upgrading" means pulling the main branch and re-reading the update log for entries that touch your tree. Config formats and model definitions can change between entries; the 2023.11.05 note about changing the logic of remap_mscoco_category, for example, is exactly the kind of change that alters how custom-dataset training behaves without changing your code.

On licensing: the repository carries Apache-2.0, confirmed by the LICENSE file at the root and the badge in the README. Apache-2.0 is permissive and includes an explicit patent grant, which matters for a model implementation. Two things to check yourself rather than assume. First, whether the pretrained weights you download carry the same terms as the code, since the README links weights through issues rather than a model card. Second, whether the community ports listed under Implementations (the Android LiteRT project, for instance) are separately licensed. This is a description of the licence file, not legal advice.

## Conclusion

Adopt RT-DETR if you need a DETR-family detector with published COCO numbers and you are willing to clone the repository and pick between rtdetr_pytorch, rtdetrv2_pytorch, rtdetr_paddle and rtdetrv2_paddle yourself. Do not adopt it if you want a pip install, a stable inference API, or a maintained single package; the README documents training and export paths but not a supported Python entry point. Before committing, verify which of the four trees matches the weights you intend to use, check that your target deployment path (ONNX Runtime, TensorRT or OpenVINO, which the README mentions only through a linked discussion) is covered by an export script in that tree, and confirm the Apache-2.0 licence file at the repository root covers the code you copy.

## FAQ

### What does RT-DETR stand for?

Real-Time DEtection TRansformer. The repository title spells it out as "RT-DETR: DETRs Beat YOLOs on Real-time Object Detection", and the search phrase "rt-detr (real-time detection transformer)" reflects the same expansion.

### Is RT-DETR open source?

Yes. The repository carries an Apache-2.0 licence, shown by the LICENSE file at the root and the licence badge in the README, and the full source for all four implementations is in the repository.

### Is RT-DETR a vision transformer?

It is a detection transformer, which is the DETR family the name refers to. The README does not describe the encoder and decoder internals in the text, so read the linked papers (arXiv 2304.08069 and arXiv 2407.17140) for the architecture rather than the README.

### Is RT-DETR better than YOLO?

The paper title makes that claim, and the README's COCO table gives RT-DETR accuracy, parameter, FLOPs and T4 TensorRT FP16 frame-rate figures so you can compare them against YOLO numbers you trust. The comparison only holds if the input resolution and evaluation protocol match, and the README gives no YOLO results of its own.

### How do I use RT-DETR?

The README points at four directories: rtdetr_pytorch, rtdetrv2_pytorch, rtdetr_paddle and rtdetrv2_paddle, each holding code and weights. For inference without training, the README also notes that RTDETR and RTDETRv2 are available in Hugging Face Transformers and in ultralytics/ultralytics.

## Sources

- [Issues](https://github.com/lyuwenyu/RT-DETR/issues)
- [License: Apache-2.0](https://github.com/lyuwenyu/RT-DETR/blob/main/LICENSE)
- [lyuwenyu/RT-DETR on GitHub](https://github.com/lyuwenyu/RT-DETR)
- [README](https://github.com/lyuwenyu/RT-DETR/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/lyuwenyu-rt-detr