SpatialClaw makes the model write Python in a kernel that remembers, instead of calling tools once

One Python cell per step, and the kernel keeps every earlier result

The interface decision is the paper. SpatialClaw's claim is that code is the right action interface for spatial reasoning agents, and it names the two designs it is replacing. Single-pass code execution commits to a full analysis strategy before any intermediate result is observed. A structured tool-call interface is flexible for calling things but poor at freely composing operations or tailoring an analysis to one task. What SpatialClaw does instead is give the model a stateful Python kernel and let it write one executable cell per step, conditioned on everything that came before.

The loop has five stages per sample. A planner drafts a strategy. The main vision-language model writes one Python cell for that step. The cell is AST-checked and then executed in the stateful kernel. Stdout, any new variables, and images produced by a show() call flow back as the next observation. Then the loop repeats, until the agent commits to an answer with ReturnAnswer(...). The AST check is the guard between the model and execution, and the runtime around it is a LangGraph workflow with planning and reflection loops rather than a single prompt and a parser.

The practical difference is that a mistake is recoverable. Because the kernel keeps state, a cell can look at what the previous cell produced, disagree with it, and continue, instead of the whole analysis being decided before the first pixel is measured.

The kernel arrives pre-loaded, which is the whole point

An empty kernel would make this a generic code agent, so the loading matters more than the loop. SpatialClaw maintains a Python kernel pre-loaded with the input frames for the sample plus a suite of perception and geometry primitives, and the shipped environment includes the scientific libraries those primitives and the model both expect. Perception comes from SAM3 for segmentation and Depth-Anything-3 for reconstruction, alongside geometry utilities, and NumPy, SciPy, and Matplotlib are available for the analysis in between.

Two details change what the agent can do compared with a tool-calling loop. First, tool outputs are composable, because a reconstruction result and a segmentation mask are just variables that a later cell can combine numerically rather than two opaque returns the model has to describe in prose. Second, show() images come back as observations, so the agent can look at a plot or a mask it just drew and revise based on what it sees, which is what makes the loop adapt to visual evidence and not only to text.

There is also a perception tool server, served over FastAPI, which hosts the GPU side of Reconstruct and SAM3. That is what keeps the heavy models out of the kernel process, and it is why the runtime is described as three independent services rather than one program.

Three services coordinate, and a cluster is optional

At runtime there are three independent services: a vLLM backbone serving the language model, the GPU perception-tool server for Reconstruct and SAM3, and the agent itself, which is a set of Jupyter kernels. They are not one process. They coordinate through shared JSON registries, and on a cluster they survive job time limits through auto-restarting chain jobs, so a long benchmark run can outlive the wall clock of the allocation that started it. The backbone side includes vLLM auto-discovery and load balancing, which is how one logical model name maps onto more than one replica.

The important qualification is that SLURM is not required. Each of the three services is also a plain Python entry point that can be run on any GPU machine, and the quickstart is written for exactly that case, describing a single machine with no SLURM. The launch managers exist for reproducing the paper's tables, not for first contact with the code.

Monitoring is treated as its own concern rather than a logging side effect. There is a dedicated guide for dashboards, SLURM logs, per-sample outputs, and stopping services, and a separate troubleshooting guide for common errors and fixes. That split is a signal about the shape of the system: when a run fails you need to know which of the three services died, and a single log tail rarely tells you.

The quickstart names a Gemini config while the shipped configs point at vLLM

Setup is a clone with submodules, a setup script, and a copy of the environment file:

git clone --recursive https://github.com/NVlabs/SpatialClaw.git
cd SpatialClaw
bash spatial_agent/scripts/setup.sh

The recursive flag is not optional, because the repository carries a .gitmodules file and the installation guide lists third-party submodules among its prerequisites. The example environment file is copied and edited:

cp .env.example .env

Then an experiment runs through one module with three flags:

python -m spatial_agent.entrypoints.run \
    --dataset spatial_agent/config/dataset/erqa.json \
    --model   spatial_agent/config/model/gemini-3-pro.json \
    --concurrency 4

Here is a detail that costs people an afternoon. The model config in that command is named for Gemini, but the example environment file states that the shipped configs all use self-hosted vLLM with a bearer placeholder and do not need a key. NVIDIA_API_KEY exists for the model configs whose llm_base_url points at a hosted OpenAI-compatible endpoint, and OPENAI_API_KEY is only a fallback for the case where llm_api_key resolves to an empty string. Values are pulled in by get_config() and referenced from any model config JSON through dollar-brace variable expansion, and anything already exported in the shell wins over the file. The file itself is git-ignored.

Every config field has an environment variable override

Configuration is JSON files plus environment variables, and the override is total rather than partial. Any dataclass field on SpatialAgentConfig can be replaced by setting a variable named after it with a fixed prefix, and and the example file spells out two of them:

SPATIAL_AGENT_BENCHMARK=erqa
SPATIAL_AGENT_CONCURRENCY=8

The naming convention is the only documentation required, which is convenient for a cluster where you would rather set an environment variable in a job script than edit a JSON file that twenty jobs share. Model configs live under spatial_agent/config/model/ and dataset configs under spatial_agent/config/dataset/, so the same split governs both what answers the questions and which questions get asked.

The documentation set is organized around those two files rather than around the paper. One guide covers prerequisites, conda and vLLM environments, submodules, and API keys. Another covers SLURM setup, pre-downloading weights, the launch managers, the direct command line, and reproducing the paper's tables. A third covers the model and dataset JSON configs, the environment overrides, and the supported benchmark list. A fourth covers the agentic loop, the three-service runtime, and the directory structure. Pre-downloading model weights is called out as mandatory before SLURM runs, and a HuggingFace token is required for gated repositories such as SAM3.1 during the download.

All 20 benchmarks ship as configs, and the category table stops after one row

Every benchmark from the paper is present as a ready-to-run dataset config, which is the part of the repository that most affects whether you can check the reported numbers yourself. The abstract describes the set as spanning a broad range of static and dynamic 3D and 4D spatial reasoning tasks, and the count is fixed at 20 with all 20 loaders included in the repository alongside the agent runtime and the tool wrappers.

The table that categorizes them does not survive to the end of the file as published. It opens with a single-image spatial reasoning row listing ERQA, Omni3D, OmniSpatial, and SPBench, and then the visible documentation ends mid-row, so the remaining categories are not readable here. That is a real limit on what can be said about the coverage split between single-image, multi-view, video, and dynamic scenes, and the supported benchmark list in the configuration guide is where a complete enumeration belongs.

The practical consequence for anyone planning a run is that you should treat the dataset directory rather than the overview table as the source of truth for what is available, since the configs are what the entry point actually consumes.

59.9 percent average with one system prompt across six backbones

The reported result is a 59.9 percent average accuracy across the 20 spatial reasoning benchmarks, which the project says beats the prior best spatial agent by 11.2 points. The claim that carries the weight is not the average but the conditions: the same system prompt, the same tool set, and the same hyperparameters across all benchmarks, with no benchmark-specific or model-specific adaptation, and consistent gains across six vision-language backbones from two families, named as Qwen3.5 and 3.6 and Gemma4, spanning 26B to 397B parameters. The framework is training-free, so nothing here is attributable to fine-tuning; the difference being claimed comes from the interface.

Those are the project's own numbers, stated in its own abstract and summary, and nothing in the repository names an independent reproduction, so they are worth reading as a claim with a specified protocol rather than as a settled measurement. What makes the protocol unusually checkable is that the protocol is also shipped: the same system prompt, the same tool set, and the same dataset configs are in the repository, so the comparison can be rerun rather than taken on faith.

Two smaller facts frame the project itself. The repository is the official implementation of the paper, with the authors from NVIDIA and KAIST AI and the work done during the first author's internship at NVIDIA, and a PDF of the paper is linked from the project page. There are no GitHub releases, so the main branch is the only version, and the license metadata field carries no explicit grant even though a LICENSE file sits at the repository root.

Editorial conclusion

SpatialClaw is worth reading if you build vision-language agents and have hit the ceiling that comes from a structured tool-call interface, because the stateful kernel plus an AST check is a concrete alternative you can copy, and the repository ships the loaders, the tool server, and the launch managers rather than a paper and a stub. It is a poor fit for a single machine with one consumer GPU, since the quickstart assumes agent and vLLM environments, submodules, and pre-downloaded weights, and the three-service runtime is designed around GPU memory. Before trusting the reported numbers, check three things yourself: they are the project's own evaluation with no independent reproduction named here, they hold a single system prompt and one hyperparameter set across 20 benchmarks, and the license field in the repository metadata carries no explicit grant, so read the LICENSE file before you build on it.

Frequently asked questions

Official sources

1. Issues
2. NVlabs/SpatialClaw on GitHub
3. Project website
4. README