WildDet3D turns text, point and box prompts into 3D boxes

One detector driven by text, point or box prompts

WildDet3D is described as scaling promptable 3D detection in the wild, and the project page, paper, model card, dataset, HuggingFace collection and blog post are all linked from the README. The practical meaning of promptable here is visible in the HuggingFace interactive demo, which is described as an interactive web demo with text, point and box prompts. A text prompt names an object, a point prompt clicks a location in the image, and a box prompt draws a region, and the model returns 3D detections. The point of the three prompt types is that they share one model rather than being three separate systems, so a robotics pipeline that knows roughly where something is and a human annotator who only knows what something is can use the same detector. The model weights sit on HuggingFace under the allenai organisation and the accompanying dataset is published as WildDet3D-Data.

Six demo surfaces, from a browser to a robot arm

The repository ships a demo/ directory with a separate subproject for each surface, and the README table is the fastest way to see the scope. There is the HuggingFace interactive demo with a live space and a local run guide, and an iPhone app offering real-time on-device 3D detection, published on the App Store with its own video and README. There is an integration demo for combining the detector with vision-language models, which is aimed at pipelines where a language model decides what to ask about the scene. There is a zero-shot tracking demo for 3D object tracking without training, which extends detections into a trajectory rather than a single frame. Then there is Meta Quest support for 3D detection in AR and VR, and a robotics demo for 3D perception in robotic manipulation. Each demo has its own README, so the practical entry point is picking the one closest to your hardware rather than installing the whole set.

Replacing OWL and BoxerNet inside Meta FAIR's Boxer

The most concrete deployment story in the README is the integration with Meta FAIR's Boxer demo for indoor labelling. The claim is that WildDet3D replaces OWL and BoxerNet inside the Boxer indoor labelling pipeline on Project Aria, and that Boxer's AriaLoader, offline fusion, online tracker and 3D viewers all run on WildDet3D's outputs unchanged. That last clause is the one that matters technically, because it says the swap happens upstream of everything else in the pipeline rather than requiring changes throughout it. Boxer's own role is as a labelling tool fed by Aria camera data, and the components named, data loading, offline fusion, online tracking and the viewers, are untouched. For a team already running such a pipeline, the interesting question is not whether the model works but whether the outputs land in the same data structures the existing fusion and tracking code expects.

The install sequence starts with a CUDA pinned torch

Getting the environment right is the first practical hurdle, and requirements.txt documents the order explicitly. The comment at the top says to install torch, vis4d and vis4d_cuda_ops before running pip install -r requirements.txt, and points to the README installation section for the full sequence. The wheel index matters, because the pinned torch build comes from the PyTorch download index rather than plain PyPI:

#   pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121

The requirements file then lists the pinned versions as a soft floor, torch 2.5.1 and torchvision 0.20.1, with numpy and Pillow as the core additions. Model dependencies include einops, timm and transformers, plus huggingface_hub for pulling the checkpoint. Note the indentation on that pip line: it sits inside a comment block, which tells you it is guidance rather than a command the file runs for you.

utils3d comes from GitHub, and sam3 comes from a submodule

Two dependency details will save an evening if you read them first. utils3d is listed with a git reference, and the comment explains why: the PyPI package utils3d at version 0.1.0 is an unrelated stub, while the real project under EasternJournalist exposes utils3d.pt and utils3d.np helpers and only ships from GitHub. So a plain requirements install that resolves utils3d from PyPI will fail later at import time rather than at install time. The second detail is sam3, which the comment describes as a third_party dependency loaded through sys.path rather than pip installed, with its own transitive requirements of ftfy, regex, iopath and typing_extensions. It also pins setuptools below 80, because a model builder module still imports pkg_resources, which setuptools removed. The repository has a .gitmodules entry, which is consistent with sam3 arriving as a submodule.

Evaluation configs for Omni3D, ScanNet and Argoverse 2

The news section records that on 2026-05-20 the project released evaluation configs for Omni3D, ScanNet and Argoverse 2, covering text and box prompts crossed with monocular and ground truth depth inputs, four modes each. Reading that as a matrix, it gives twelve evaluation configurations across three datasets, which tells you two things about the project's design. First, both prompt types are treated as first class for evaluation rather than text being the default and box being an afterthought. Second, depth is a variable input, so the same detection model is evaluated with the monocular path and with depth supplied, which is the difference between a depth dependent geometry stage and one that can infer it. The repository carries a configs/ directory, plus scripts/ and docs/ directories, and the dataset card WildDet3D-Data on HuggingFace is where the data side is published.

No releases, a custom licence and an iPhone app outside the tree

Several practical adoption details sit outside the demos. The repository publishes no GitHub releases, so there is no tagged version to pin and installation comes from the source tree plus the published model weights. The licence is a custom one that GitHub cannot classify, so the LICENSE file in the repository is the authoritative text and the README badge is not a substitute for reading it. The iPhone app with real-time on-device detection is distributed through the App Store rather than built from this repository, which tells you the on-device path is a separate artifact with its own release cycle. The last push was on 2026-06-01, and the demo for Meta Quest and the robotics demo both rely on external toolchains described in their own READMEs, so treat each surface as its own project with its own setup cost rather than one install that covers everything.

Editorial conclusion

WildDet3D fits a team that needs 3D boxes from ordinary camera images without training a detector, and a labelling team that wants to replace an open-vocabulary detection plus geometry pipeline with one model. It does not fit a project that needs a packaged release, since the repository publishes no GitHub releases and carries a custom licence whose terms you have to read in the LICENSE file, and it does not fit a machine without a CUDA setup given the pinned torch wheel index. Verify first that you can install torch 2.5.1 with the cu121 wheel, that the utils3d dependency resolves from GitHub rather than PyPI, and that the demos you care about are covered by the evaluation configs released on 2026-05-20.

Frequently asked questions

Official sources

1. allenai/WildDet3D on GitHub
2. Issues
3. Project website
4. README