AViD: fine-tuning Grounding DINO on your own objects with LoRA

What AViD adds on top of Grounding DINO

Grounding DINO is an open-vocabulary detector: it grounds text phrases in images. AViD is a training framework around it that turns that general capability into something specific to your own objects. The README describes the aim plainly, teams that need domain-specific grounding without updating every parameter of a large detector.

The scope is a complete loop rather than a training script. CSV dataset loading, training, checkpointing, validation and standalone evaluation all live in one repository. Parameter adaptation uses LoRA, and the README states the targets explicitly: the detector's attention, projection, feed-forward, bounding-box and feature-map layers, with rank 32 as the default starting point. Training stabilisation is optional, through exponential moving averages that are meant to retain a smoother model trajectory during fine-tuning.

Entry points are deliberately plain. YAML drives training and evaluation, a script handles single-image inference, and a Gradio app covers interactive inspection. The evaluation pipeline reports mAP, precision, recall, F1 and per-class summaries, and can produce visual comparisons of predictions.

One point deserves to be read twice, because it shapes what this project can and cannot claim for itself. The README states that AViD does not yet publish a checkpoint, an immutable dataset revision and a machine-readable run manifest capable of supporting a verified benchmark table, and that no numerical result is claimed as a result. What it offers instead is an evaluation protocol document, and every new JSON report records the vocabulary and thresholds so that future results can be compared honestly. That is a project being careful about reproducibility rather than one that has results to sell.

How the dataset reader turns CSV rows into a class vocabulary

AViD's training dataset reader expects one CSV row per object, with the image name, the bounding box in pixel coordinates and a label.

image_name,bbox_x,bbox_y,bbox_width,bbox_height,label_name
example.jpg,120,80,240,310,shirt
example.jpg,410,190,160,120,bag

The box format is x, y, width, height in pixel space. Image names are resolved relative to the configured image directory, so the CSV itself holds no absolute paths. Labels are normalized to lowercase, and AViD builds one sorted class vocabulary that is shared by every training and validation image. That single vocabulary is the contract between the two splits, and it is also why label casing in the CSV matters more than it might seem.

The configuration file points the loader at the two splits separately, with a train directory, a train annotation file, a validation directory and a validation annotation file, plus the batch size and the number of data loader workers. The default in the shipped example is a batch size of 4 with 8 workers, which is a small-batch setup appropriate for the fine-tuning regime rather than for training a detector from scratch.

The README also points at a sample fashion subset that can be fetched with gdown and unzipped into a directory named multimodal-data, which gives a new user a working dataset to check the pipeline against before labelling their own images.

Installing AViD: the default CPU build and the separate CUDA build

The default installation is a portable Python package install that supports CPU execution.

git clone https://github.com/levyflux/AViD.git
cd AViD
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

GPU execution is a separate path, because AViD inherits a custom CUDA operator from Grounding DINO. The order matters: install a PyTorch build compatible with your CUDA toolkit first, then build the operator explicitly without build isolation. Two environment variables drive that build. TORCH_CUDA_ARCH_LIST tells the toolchain which GPU architecture to target, with the example value 8.6 that the README asks you to replace with your own compute capability, and AVID_BUILD_CUDA set to 1 switches the CUDA extension on. You can check what you have with nvidia-smi before setting them.

The failure behaviour here is worth noting because it is unusual in a good way. The README states the CUDA build fails closed when PyTorch, nvcc or CUDA_HOME is unavailable, instead of silently producing a GPU installation without its required operator. A silent success in that situation would produce a model that appears to train and then falls over at inference, so the explicit failure is the friendlier option.

The dependency list in requirements.txt shows the stack in use: torch and torchvision, transformers, timm, peft for the LoRA layers, ema_pytorch for the averaging, supervision pinned at 0.6.0, pycocotools, opencv-python, scipy, numpy, matplotlib, tqdm, pillow, addict and yapf. The demo has its own requirements file, requirements-demo.txt, so the Gradio app can be installed separately from the training stack.

The YAML configuration surface and what a single run leaves behind

Training is driven by a configuration file, and the shipped example is compact enough to read in one pass. The data block holds the paths and the loader settings, the model block holds the Grounding DINO configuration path and the checkpoint path, and the training block holds the schedule.

data:
  train_dir: multimodal-data/fashion_dataset_subset/images/train
  train_ann: multimodal-data/fashion_dataset_subset/train_annotations.csv
  val_dir: multimodal-data/fashion_dataset_subset/images/val
  val_ann: multimodal-data/fashion_dataset_subset/val_annotations.csv
  batch_size: 4
  num_workers: 8
model:
  config_path: groundingdino/config/GroundingDINO_SwinT_OGC.py
  weights_path: weights/groundingdino_swint_ogc.pth
training:
  num_epochs: 200
  learning_rate: 1e-3
  use_lora: true
  use_ema: false

The defaults in that example are worth reading as a statement of intent: 200 epochs, a learning rate of 1e-3, use_lora set to true and use_ema set to false. The model section points at the Swin-T configuration file inside the vendored groundingdino directory and at a checkpoint under weights. The README explains where that checkpoint comes from, the Grounding DINO Swin-T configuration and checkpoint pair downloaded from the upstream project, and where it has to be placed for the weights_path setting.

The command itself takes one required argument.

python train.py --config configs/train_config.yaml

Two flags matter for practice. `--save-dir` overrides the configured parent directory for outputs, and `--device cpu` runs a smoke test on CPU without a GPU present. Each invocation creates one timestamped run directory containing the reloadable configuration, checkpoints, history, visualizations and evaluation outputs. Storing the configuration inside the run directory is the detail that makes a later comparison possible, since the run carries its own settings rather than relying on the file you edited this morning.

The evaluation protocol: fixed vocabulary, global ranking, recorded thresholds

Evaluation is a separate script rather than a flag on training, and it takes a configuration, an output directory, a device and an optional visualisation switch.

python evaluate.py \
  --config configs/evaluation_config.yaml \
  --output-dir evaluation_results \
  --device cuda \
  --visualize

The protocol is specified rather than left to implementation. AViD uses one fixed vocabulary for the entire evaluation split, and predictions are ranked globally per class across the dataset rather than per image. Missing detections and false positives stay in the metric instead of being filtered out. The README is explicit that this is a phrase-conditioned detection protocol and not COCO mAP, and every metrics.json records the protocol, the class names, the IoU thresholds, the score threshold and the maximum detections.

That is a narrower and more honest claim than a mAP number with no protocol attached. It also has a practical consequence for anyone comparing AViD against another detector: the numbers are only comparable to another run that used the same vocabulary and the same thresholds, which is exactly why the report stores them. The evaluation-during-training configuration is documented in EVALUATION.md alongside the metrics and the output layout.

The visual comparison feature is part of the same pipeline, and it matters more than a table of numbers would, because a phrase-conditioned protocol is easiest to audit visually. The qualitative before and after images in the README come from the original fashion experiment, and the surrounding text is careful to call them examples rather than results.

Single-image inference and the interactive demo

The inference script is the fastest way to see what a checkpoint does. It takes the Grounding DINO configuration path, a checkpoint path, an image, a text prompt, an output directory and two thresholds.

python demo/inference_on_a_image.py \
  --config_file groundingdino/config/GroundingDINO_SwinT_OGC.py \
  --checkpoint_path weights/groundingdino_swint_ogc.pth \
  --image_path .asset/cat_dog.jpeg \
  --text_prompt "cat . dog" \
  --output_dir outputs \
  --box_threshold 0.30 \
  --text_threshold 0.25

The prompt is a phrase list in the Grounding DINO style, where a dot separates the phrases, and the thresholds control the box and text scores. There is also a phrase-specific mode, where `--token_spans` takes a nested list of character ranges into the prompt string. The README makes a point of the safety properties of that argument: it is parsed as data and validated, it is not executed as Python. That is worth knowing because passing a nested list on a command line is exactly the kind of interface that invites a naive implementation using eval.

The Gradio demo is separate and pulls the upstream Grounding DINO checkpoint from Hugging Face on first use. It has its own requirements file and is started with a script under demo. The README notes that the `--share` flag should be added only when you deliberately want Gradio to create a public share link, which is a sensible default for a tool that can expose an interactive interface to the internet.

The demo directory also holds a script for creating a COCO-style dataset, a test script for average precision on COCO, and two notebooks on image editing with Grounding DINO and Stable Diffusion. Those extras sit outside the training workflow but explain what the project expects from the surrounding ecosystem.

Limitations, the licence picture and when full fine-tuning is the better answer

The largest limitation is already stated by the project. There is no published checkpoint, no immutable dataset revision and no machine-readable run manifest, so there is no verified benchmark table and no numerical claim. A team that wants a working custom detector today has to train it, and the example dataset that ships with the project is a fashion subset rather than a domain corpus for your business. The second limitation is operational: the GPU path needs a matching PyTorch build, a correct compute capability value and an explicit operator build, and a mismatch there fails the install rather than degrading quietly.

The licence picture has two parts that a reader should see separately. The repository is MIT licensed, and the badge in the README points at the MIT licence file. The setup.py file, however, carries an Apache License 2.0 header naming Levy Hu and describing itself as modified from Deformable-DETR, detectron2, mmdetection and libai, and the vendored groundingdino directory is upstream Grounding DINO code. Those are separate copyright statements covering separate files, and anyone redistributing a build should read both rather than assume a single licence covers the tree. This is a reading of the file headers, not legal advice.

The real alternative to consider is full fine-tuning of Grounding DINO rather than LoRA adaptation. The difference in approach is direct. AViD's LoRA path updates attention, projection, feed-forward, bounding-box and feature-map layers through low-rank adapters at rank 32 by default, so the trainable parameter count and the checkpoint size stay small and a run fits on modest hardware. Full fine-tuning updates every parameter of the detector, which gives the model more freedom to fit a domain that is far from the pretraining distribution, and it costs more memory, more storage and a longer schedule. If your objects barely resemble anything in the pretraining data, the parameter-efficient path may cap how far the detector can move, and the project does not publish evidence about that boundary.

On maintenance, the repository is not archived and the last push was on 2026-08-30. The project is young and its own documentation is explicit about what it has not yet produced, which is a reasonable basis for adoption if you intend to run the training yourself. The tooling around it is conventional: ruff is configured in pyproject.toml for a Python 3.10 target with a line length of 100, pytest collects from the tests directory, and the repository includes AGENTS.md, CLAUDE.md and CONTRIBUTING.md alongside the code.

Editorial conclusion

AViD suits teams that already work with Grounding DINO, have a labelled image set, and want a repeatable training and evaluation loop without retuning every parameter of a large detector. It does not suit anyone who needs a pretrained custom checkpoint today, because the project publishes none. Verify first that your GPU compute capability matches the TORCH_CUDA_ARCH_LIST you set, that the base Swin-T checkpoint is in place at model.weights_path, and that your CSV labels form the vocabulary you intend to evaluate against.

Frequently asked questions

Official sources

1. Issues
2. levyflux/AViD on GitHub
3. License: MIT
4. README