# RelateAnything: relation prediction that takes your regions and your vocabulary

> RelateAnything is a model that turns an image plus any set of regions into scored subject, relation, object triplets, and the two things that make it different from a fixed label set are that the regions can come from any detector and that the relations you look for are supplied at inference time. It runs on PyTorch, on TensorRT, on ONNX and OpenVINO, and in a browser through WebGPU.

**Maelic/RelateAnything** — RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs 

- Repository: https://github.com/Maelic/RelateAnything
- Website: https://maelic.github.io/RelateAnythingProject/
- Stars: 1,146 · Forks: 159
- Language: Python
- License: AGPL-3.0
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/maelic-relateanything

## Regions in, triplets out, and the vocabulary is yours

RelateAnything is described as real time open vocabulary relation prediction from any inputs, and the usage model in the README is short. You give the model an image, object regions, and the relations you want to look for, and it returns scored subject, relation, object triplets, such as `person` riding `horse`. Three properties follow from that sentence. The regions are yours. Boxes can come from a detector, a segmenter, or a person, masks are optional, and the model's inputs are an `[N, 4]` array in original image pixels, so the detector's labels are never an input and the labels in the example are for display only. The relations are yours too, since you supply phrases at inference time without retraining. And the output shape is yours, because you can get one ranked list or two separate graphs, a spatial one describing where things are and a semantic one describing what they do, produced from the same forward pass. The decomposition is the part that shows the design is not a repackaging of a fixed scene graph model, because separating where from what is a modelling decision the trained model has to support.

## Why there is no language model in the loop

The README makes a point about efficiency that is worth repeating, that a small text encoder embeds the relation phrases once and no language model runs per frame. That is the technical decision that makes the open vocabulary claim affordable. The usual way to make a vision model accept free text is to run a language model alongside it on every frame, which costs memory, latency and a lot of engineering, and which would make the real time claim in the project tagline hard to keep. Here, the phrases you care about are embedded a single time, when you set the vocabulary, and the relation head then works in that embedded space. The consequence for a user is a workflow where the vocabulary is a configuration decision rather than a training decision. You decide the set of relations your application cares about, set it once, and the same weights serve every vocabulary you can think of. It also explains the choice to have `set_vocabulary` as a separate call from `predict`, and it is why the deployment section can promise that vocabulary selection stays available in the TensorRT engine rather than being baked in at build time.

## Getting the first triplets out in a few lines

The quickstart needs Python 3.12 or later, and either a CUDA enabled PyTorch build for NVIDIA GPU inference or `device="cpu"` in the example. Installing from the repository is a clone and an editable install with two extras, the `hub` extra for pulling weights from the Hugging Face Hub and the `detector` extra, which installs ultralytics for the YOLO detector used in the example.

```bash
git clone https://github.com/Maelic/RelateAnything.git
cd RelateAnything
pip install -e ".[hub,detector]"   # detector: ultralytics YOLO, used below
```

The example is three steps, find the objects, predict the relations, and then change the vocabulary without retraining:

```python
from PIL import Image
from ultralytics import YOLO
from relsgg import RelateAnything

image = Image.open("assets/reel/images/horse.jpg").convert("RGB")

# 1. Find the objects. Any detector or segmenter works; this one is YOLO26.
detections = YOLO("yolo26m.pt")(image, conf=0.4, verbose=False)[0]
boxes = detections.boxes.xyxy.cpu().numpy()  # [N, 4]: x1, y1, x2, y2 in pixels
labels = [detections.names[int(c)] for c in detections.boxes.cls]  # for display only

# 2. Predict the relations between them.
model = RelateAnything.from_pretrained("maelic/relsgg-vits16plus", device="cuda")
for triplet in model.predict(image, boxes, box_labels=labels, topk=3):
    print(triplet)
```

The printed output in the README is the shape you should expect, with a subject, a relation and a confidence, and a second call sets a new vocabulary and asks for the decomposed graphs with `decompose=True`, giving back the `spatial` and `semantic` graphs. One point deserves attention: the README states that the released checkpoints include the backbone configuration and text encoder and that no gated DINOv3 login is needed for inference, which removes the usual friction of running a model whose backbone sits behind a licence request.

## Seven checkpoints, and a measured speed and accuracy trade-off

The model family is published as seven checkpoints that share the training recipe and differ in their vision backbone, and the README gives the parameter counts. The smallest is relsgg-vits16 on a DINOv3 ViT-S/16 backbone at 46.1 million parameters, then relsgg-vits16plus on ViT-S/16+ at 53.2 million, which is the recommended balance of quality and speed, then relsgg-vitb16 on ViT-B/16 at 113.8 million. The ConvNeXt family adds relsgg-convnext-tiny at 58.6 million, convnext-small at 80.2 million, convnext-base at 123.5 million and convnext-large at 245.7 million. Two details about the training are worth knowing if you plan to fine tune rather than just infer. The ConvNeXt towers also train their backbone at a lower learning rate, and for more accuracy the README points at relsgg-convnext-base, which is within 0.3 OVS of ConvNeXt-L at half its parameters. The accuracy metric used throughout is OVS-F1 across four benchmark axes, and the README includes a plot of that against latency on a laptop RTX 3080 with TensorRT, where the ConvNeXt-T, ViT-S+, ConvNeXt-S, ConvNeXt-B and ConvNeXt-L checkpoints form the Pareto frontier and ViT-S and ViT-B sit below it.

## Four deployment paths, and they are not the same amount of work

The deployment section lists four routes, and the differences between them matter for planning. With PyTorch you use the Python API on CPU or CUDA, which is the path the quickstart already covers. With an NVIDIA GPU and TensorRT 10 you build a local FP32 engine from the released ONNX graph, and the README notes that vocabulary selection, calibration and the two graph decoding stay available in that path. On a laptop, with ONNX or OpenVINO, you run the local image or webcam demo. In the browser you try the demo without installing Python, because inference runs on your device using WebGPU or WASM. The browser route is the one that changes the shape of a product, since a user can try the model without a Python environment, and it is also the one with the most moving parts, since the model has to be shipped as a browser friendly bundle. The documentation splits these out properly, with a deployment guide for the ONNX and OpenVINO laptop path, a deployment README with the TensorRT setup, and an API guide for inputs, vocabularies, masks and scores.

## Pinned dependencies, a licence that matters, and the paper trail

The package metadata is unusually careful, and two things in it deserve a reader's attention. The dependency pins are written as compatible release constraints, and the comment explains why, that they are pinned against the environment every released checkpoint was trained and verified with, that Python 3.13 and that numpy 2.5 needs 3.12 or later, and that loosening a pin is fine for experimentation while the pins document what is known to reproduce. The second is the comment on pycocotools, which notes that under plain pip it resolves to the upstream package, and that the fork in the project's own repository is only picked up by uv, with the pip install command spelled out for users who need the fork. The extras are laid out by purpose, with a `deploy` extra for the ONNX demo and laptop runtime, a `tensorrt` extra for native TensorRT 10 engine builds, a `gradio` extra for the server side demo and a `hub` extra for the Hugging Face Hub. The licence is AGPL-3.0, and the README notes that the detector extra installs ultralytics, which is also AGPL-3.0, so a deployment that ships the detector inherits two copyleft licences. The repository carries a NOTICE file, a THIRD_PARTY_NOTICES.md, a CITATION.cff and a CONTRIBUTING guide, and the paper is on arXiv.

## Conclusion

Adopt RelateAnything if you need scene graph style relations over regions you already have, and you want to choose the relation vocabulary at run time rather than train for a fixed label set. Do not adopt it into a closed source product without reading the AGPL-3.0 terms, and note that the detector extra pulls in ultralytics, which is also AGPL, so the licence question arrives twice. Before you build, decide which checkpoint you need, because the family spans 46 million to 245 million parameters with a measured speed and accuracy trade-off, and start with the recommended relsgg-vits16plus. Also plan the deployment path early, because the browser and TensorRT routes are separate pieces of work from the Python API. The licence is AGPL-3.0, the package version is 1.0.0, the last push was on 2026-10-12, and the repository publishes no GitHub releases.

## FAQ

### What does RelateAnything take as input?

An image, object regions, and the relations you want to look for. Boxes can come from a detector, a segmenter or a person, masks are optional, and an `[N, 4]` array in original-image pixels is all it needs. The detector's labels are never an input, only for display.

### Can I change the relation vocabulary without retraining?

Yes. You supply the relation phrases at inference time and call `set_vocabulary`, after which a small text encoder has embedded them once. The README stresses that no language model runs per frame, and the TensorRT deployment path keeps vocabulary selection available.

### Which RelateAnything checkpoint should I start with?

The README recommends relsgg-vits16plus as the balance of quality and speed, at 53.2 million parameters on a DINOv3 ViT-S/16+ backbone. For more accuracy it points at relsgg-convnext-base, which is within 0.3 OVS of ConvNeXt-L at half its parameters.

### How do I install and run RelateAnything?

Use Python 3.12 or later, clone the repository, and run `pip install -e ".[hub,detector]"`. Then load the model with `RelateAnything.from_pretrained("maelic/relsgg-vits16plus", device="cuda")` or `device="cpu"` and call `predict` with your image, boxes and optional labels. The `detector` extra installs ultralytics for the YOLO detector used in the example.

### Can I run RelateAnything in a browser?

Yes. The hosted demo runs inference on your device using WebGPU or WASM, with no installation. Other routes are the Python API on CPU or CUDA, a local FP32 TensorRT 10 engine built from the released ONNX graph, and the ONNX or OpenVINO laptop demo.

## Sources

- [Issues](https://github.com/Maelic/RelateAnything/issues)
- [License: AGPL-3.0](https://github.com/Maelic/RelateAnything/blob/main/LICENSE)
- [Maelic/RelateAnything on GitHub](https://github.com/Maelic/RelateAnything)
- [Project website](https://maelic.github.io/RelateAnythingProject/)
- [README](https://github.com/Maelic/RelateAnything/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/maelic-relateanything