DAAAM turns a camera stream into a 3D dynamic scene graph

The pipeline from pixels to a spatial graph

The README describes DAAAM as real-time, foundation-model-first robot mapping, and the pipeline is three stages feeding a fourth. SAM performs segmentation, BotSort performs tracking, and a vision language model performs grounding. All three feed into Hydra, the MIT-SPARK library, and the output is 3D dynamic scene graphs built on the fly. The name stands for Describe Anything, Anywhere, at Any Moment. The three listed contributions are a novel optimization-based frontend for semantic descriptions from localized captioning models, hierarchical 4D scene graph construction with real-time performance, and state-of-the-art results on the NaVQA and SG3D benchmarks. The frontend is the interesting part conceptually, since it inverts the usual direction: rather than asking a model to caption a crop, it optimizes a description so that the geometry of the scene constrains what can be said about each region.

Dynamic means the graph changes while you watch

A scene graph is a way of writing down what is where and what is related to what, and a dynamic one keeps rewriting it as the scene changes. Combined with the 4D construction, the graph carries time as a dimension rather than being a snapshot. That is the capability the whole assembly of models is aimed at: a robot that needs to remember not only that a chair is in the room but where it was and whether it has moved. The README claims real-time performance for the hierarchical construction, and that claim is worth reading against the dependency list, because segment_anything, sam2, ultralytics, a tracking model, a captioning model and a language model all have to run inside the loop. The Hydra library is what makes the 4D graph tractable, since the scene graph representation and the optimisation over it live there rather than being rebuilt in this repository.

The dependency stack is the real specification

requirements.txt is more informative than the README about what this thing actually needs, and it is a long one. The learning stack is pinned to a CUDA 12.8 build:

torch==2.8.0+cu128
torchvision==0.23.0+cu128
torchaudio==2.8.0+cu128

Three dependencies are installed straight from Git rather than from an index, and they are the ones that carry the method: boxmot from the author's own fork, dam from describe-anything-batch, and perception_models. Everything else is the ecosystem around them. The segmentation side pulls in ultralytics, segment_anything, sam2 and open_clip_torch. The language model side pulls in langchain with the openai, anthropic and google-genai integrations, langgraph, the openai and anthropic SDKs directly, and transformers. The graph and optimisation side uses networkx pinned at 3.1 and cvxpy with scipy, which is consistent with the optimisation-based frontend. The deployment side has onnxscript, onnxslim, onnxruntime-gpu and tensorrt-cu12, plus numba, zmq, rerun-sdk for visualisation and pydantic for structured outputs.

Installing and running it, and where the documentation lives

The README is short by design and points elsewhere for the operational detail. Installation instructions are in INSTALL.md, running instructions in RUNNING.md, evaluation instructions in EVAL.md, and a code walkthrough in CODEBASE.md. The repository also carries an install/ directory, a scripts/ directory, a config/ directory, a data/ directory and a src/ tree, so the guides map onto directories you can read. The package itself is named daaam, built with setuptools, and its classifiers say Development Status 3 - Alpha, Intended Audience Science/Research, with topic classifications for artificial intelligence and robotics. There is no command sequence in the README, so the install steps live in the linked file rather than here, and the reasonable approach is to read INSTALL.md first and the requirement list above before anything else, since the pinned CUDA wheels and the three Git dependencies are the two things likely to need attention.

Language model choice is yours, and that is deliberate

The dependency list includes the openai, anthropic and google-genai packages side by side, together with langchain and langgraph, and it also includes anthropic at 0.77.0 or newer, openai, httpx and python-dotenv. That combination says the grounding and captioning step is not tied to one provider. It also introduces a cost consideration that the README does not discuss: a real-time system that captions regions on demand will be making model calls, and the choice of provider, model and prompting strategy affects both latency and cost in a loop that runs continuously. The presence of langgraph alongside langchain suggests the orchestration of the model calls is a state machine, which is the kind of thing that becomes a tuning problem in deployment. The API keys come from the environment, given python-dotenv in the list and the config/ directory in the tree.

The ROS 2 integration is a separate project

The README is explicit that the ROS 2 interface lives in DAAAM-ROS, a different repository. That split matters more than it first appears, because a robot deployment will need the ROS side and this repository is the core. It also means the two can move independently, and that a ROS user has to track two codebases. The paper itself is by Nicolas Gorlo, Lukas Schmid and Luca Carlone, with the MIT-SPARK affiliation implied by the organisation name, and it was supported by the ARL DCIST program and the ONR RAPID program. The citation is a CVPR 2026 paper with page numbers 35002 to 35013, and the arXiv preprint is 2512.00565 from 2025. The licence is BSD-3-Clause, the default branch is main, and the last push was on 2026-09-21.

Where DAAAM is the wrong tool

Three cases. The first is a production robot that needs a scene graph it can rely on, because this is alpha research code with an evaluation-oriented repository, an unversioned release history and a heavy stack of models and pinned wheels that will drift. The second is a team that does not want a language model in the loop, since the pipeline as described spends model calls on localised captions, and there is a real cost and latency to that whether you are running in simulation or on a robot. The third is a hardware budget that cannot hold the models, since the requirements imply segmentation models, a tracker, a captioning model and a grounding model running together with a TensorRT toolchain for deployment. Against building a scene graph by hand from detector output, the difference in approach is that DAAAM leans on foundation models for the semantics rather than on a fixed set of object classes, which is what lets it describe something the taxonomy never listed, and what makes it expensive and less predictable. The benchmarks to check first are the ones the README names, NaVQA and SG3D.

Editorial conclusion

DAAAM is a good fit for a robotics or computer vision researcher who needs a spatial memory of a scene built live from camera input, and who already runs the Hydra stack and has GPU hardware to spare. It is a poor fit for a production robot deployment, since the project is marked alpha in its own metadata, the ROS 2 interface lives in a separate DAAAM-ROS repository, and the runtime expects several large models and a TensorRT toolchain at once. Check three things before you start: that you have the Python 3.12 environment the project requires, since the packaging states requires-python of 3.12 or newer, that you can serve the pinned CUDA 12.8 wheels, since the requirements file points at an extra index for cu128 and pins torch 2.8.0, and that the API keys for whichever language model you intend to use are in place, because the dependency list includes the OpenAI, Anthropic and Google generative SDKs alongside langchain and langgraph rather than one fixed provider.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. MIT-SPARK/DAAAM on GitHub
4. README