DART: turning SAM3 into a real-time open vocabulary detector without training

What the framework actually converts

DART stands for Detect Anything in Real Time, and the claim in the README is a training-free framework that converts SAM3 into a real-time multi-class open-vocabulary detector. The headline measurement is 55.8 average precision on COCO val2017 across 80 classes at 15.8 frames per second for 4 classes at 1008 pixels on a single RTX 4080.

That is a large claim to make from a segmentation model, and the mechanism is what makes it interesting. SAM3 is a promptable segmentation model, and the project treats it as a component to be restructured rather than retrained. Nothing in the pipeline requires a training set, which is the whole appeal: you get a detector that accepts class names at inference time without paying for annotation or a training run.

The measured configuration is worth parsing rather than quoting as a single number. The 55.8 average precision figure is the 80-class COCO result, and the 15.8 frames per second figure is quoted for 4 classes at 1008 pixel input. Those are different configurations of the same system, and the backbone table later in the README gives separate accuracy and speed pairs for each option, so a reader comparing against their own workload should match the class count and input size rather than take the best number.

The project is a Beta, classified as Development Status 4 with the topic Scientific/Engineering :: Artificial Intelligence, and the licence field on the repository is a custom one with a LICENSE file. The last push was on 2026-10-06.

The three backbone options and what each costs

The quick start offers three backbone choices, ordered from high quality to high speed, and they coexist so you can build all three.

The ViT-H full backbone is the highest quality option at 55.8 average precision and 13.5 frames per second. The ViT-H Pruned-16 is described as the best quality and speed tradeoff, at 53.6 average precision and 19.1 frames per second, and it is built from a distilled checkpoint with 16 blocks. The RepViT-M2.3 is the fastest with usable quality, at 38.7 average precision and 30.2 frames per second.

The spread is the useful information here. Going from the full model to the pruned one costs about two points of average precision and buys roughly 40 percent more frames per second, while the student backbone more than doubles the frame rate and gives up a much larger accuracy margin, about 17 points. A workload with a fixed set of classes and strict latency would sit at the pruned option, and a workload that must detect rare classes precisely would want the full one.

The student backbones are named in the files table as distilled RepViT, TinyViT and EfficientViT models, and the weights are published on HuggingFace alongside the DART project page. So the distilled models are not something you have to train, they are downloads, which keeps the promise of the training-free framing intact even for the fast path.

Installing it on a CUDA machine

The tested environment is Windows 11 with an RTX 4080 and 16 GB, and all the commands use bash syntax. The version table pins Python 3.11 or newer, torch 2.7.0 or newer with CUDA 12.6, torchvision 0.22.0 or newer, TensorRT 10.9.0 or newer as optional for the engine export, and numpy below 2.0.

The install sequence is four steps:

# 1. Create a conda environment
conda create -n dartsam3 python=3.11 -y
conda activate dartsam3

# 2. Install PyTorch with CUDA
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

# 4. Install DART (installs all other dependencies automatically)
pip install -e .

The TensorRT step sits between those and is optional, only needed for engine export and inference with TensorRT.

There is a Windows specific note that is worth taking seriously. You should always prefix Python commands with PYTHONIOENCODING=utf-8 on Windows, because TensorRT, torch and the detection scripts print Unicode characters that the default Windows console encoding, cp1252, cannot handle.

The files table lists what the system expects to find. The SAM3 checkpoint is called sam3.pt and auto-downloads from HuggingFace on the first run, the student weights are the distilled backbones, and you supply your own test image and video. The COCO train2017 and val2017 directories are only needed for the pruning search and for evaluation respectively.

The two stage engine build

The runtime is not one model but two, and the README is explicit that the TensorRT engine is the backbone. There is a shared encoder-decoder engine, built once, and then a backbone engine you can swap.

The encoder-decoder export and build looks like this:

python -m sam3.trt.export_enc_dec --checkpoint sam3.pt \
    --output enc_dec.onnx --max-classes 4 --imgsz 1008
python -m sam3.trt.build_engine --onnx enc_dec.onnx \
    --output enc_dec_fp16.engine --fp16 --mixed-precision none

The max-classes value is where the open-vocabulary part becomes concrete. You tell the engine how many classes the text cache will carry, and the 4 here is for the 4 class demo. The 80-class setup uses a different script, build_coco_engine.py, and a different engine file with an 80 in the name.

The backbone engine is built by a separate script, and the pruned variant points at a distilled checkpoint:

# ViT-H Pruned-16 (53.6 AP, 19.1 FPS) — best quality/speed tradeoff
PYTHONIOENCODING=utf-8 python scripts/export_hf_backbone.py \
    --image x.jpg --imgsz 1008 \
    --pruned-checkpoint distilled/pruned_16blocks.pt \
    --output-engine hf_pruned16_fp16.engine

The image argument to the export script is there to run a calibration pass, which is why the COCO train2017 images are needed only for the pruning search.

The detection run then pairs the two engines, and the README stresses that all three backbones use the same encoder-decoder engine, so switching between them means swapping only the --trt argument. No retraining and no adapter flags are needed.

Running detection, video and the 80 class path

Detection on a single image is one command, and the same flags work whether you have built a TensorRT backbone or are running in PyTorch only:

python demo_multiclass.py \
    --image x.jpg --classes person car bicycle dog \
    --checkpoint sam3.pt --fast --detection-only

The --fast flag turns on batched inference with FP16 and the presence early-exit, --detection-only skips mask generation so you get boxes and scores only, and --compile accepts default, reduce-overhead or max-autotune for torch.compile.

Video inference uses demo_video.py, and the full 80-class path adds a text cache:

# Build enc-dec engine + text cache for all 80 classes
python scripts/build_coco_engine.py --checkpoint sam3.pt

# Run video
python demo_video.py --video input.mp4 --coco \
    --checkpoint sam3.pt \
    --trt hf_backbone_1008_fp16.engine \
    --trt-enc-dec enc_dec_coco_fp16_80.engine \
    --text-cache text_cache_coco.pt \
    --imgsz 1008 -o output.mp4

The text cache is the mechanism that makes open-vocabulary detection cheap enough for real time. The text embeddings for the class names are computed once and stored, so a frame costs a vision forward pass and a lookup rather than a text pass per frame. The table of contents also lists a text cache section of its own, alongside block pruning, TensorRT export for the backbone, the encoder-decoder and the student backbones, plus a separate EfficientSAM3 backbone export, COCO evaluation, benchmarks, a precision analysis and a scripts reference.

DARTF and the integer path for Jetson

The news section announces that DARTF, Detect Anything in Real Time Faster, is now available in its own directory.

DARTF runs the SAM3 ViT-H backbone in W8A8 INT8 on a Jetson AGX Orin, and the README claims it is 43 percent faster than the DART FP16 engine, which is 275 milliseconds down to 158 milliseconds per 1008 pixel frame, a 1.74 times throughput gain, at what it calls FP32 level detection quality. The COCO val2017 figures are 56.0 average precision against 56.1 for FP32, and the energy figure is 7.7 joules per frame against 13.2.

The techniques listed are exact graph rewrites, activation-aware GPTQ and custom TensorRT plugins. Quantisation is where the accuracy comes from: activation-aware GPTQ weights the rounding of each weight by the activation statistics it will see, which is why the accuracy loss is small compared with naive INT8.

The energy number is the one that matters for a handheld, and the project has clearly measured it, because a Jetson deployment is a battery deployment. The same 158 millisecond figure and the 7.7 joules per frame are the sort of numbers that let you decide whether a frame budget is realistic before you write any code.

The separate DARTF directory also means the faster engine is a separate subproject with its own README, so the two code paths are versioned and documented independently of the main DART tree.

Editorial conclusion

DART fits a computer vision team that needs open-vocabulary detection at video frame rate without collecting and training a detector, and that is comfortable with CUDA, TensorRT and a two-stage engine build. It does not fit a CPU-only deployment, and the accuracy numbers are tied to specific hardware and to the 1008 pixel input size. Before adopting it, read the benchmark tables and the precision analysis rather than the headline figure, check that your GPU and TensorRT versions match what was tested, and plan for the fact that the fastest path needs a distilled student backbone, which trades accuracy for speed in a way the README quantifies per model.

Frequently asked questions

Official sources

1. Issues
2. mkturkcan/DART on GitHub
3. README