SpatialClaw: a code-as-action-interface agent for spatial reasoning

The action interface problem SpatialClaw is built around

Tool-augmented spatial agents hand a VLM a set of perception modules (segmentation, depth reconstruction, geometry utilities) and then constrain how the model may call them. The paper's abstract frames the limitation as the action interface itself: single-pass code execution commits to a full analysis strategy before any intermediate result is visible, and a structured tool-call interface restricts how freely operations can be composed or tailored per task. SpatialClaw's answer is to let the agent write Python, one cell per step, into a kernel that already holds the input frames and the perception primitives. The intended user is a researcher running a VLM against 3D and 4D spatial benchmarks, not an application developer looking for a perception API. The claim being tested is narrow and testable: with the same system prompt, tool set and hyperparameters, does a code interface beat a tool-call interface across many benchmarks and several backbones? The README reports 59.9% average accuracy across 20 benchmarks, described as +11.2 points over the prior best spatial agent.

The five-stage loop and what flows between stages

Per sample, the README describes a five-stage loop over a persistent Python kernel. A planner drafts a strategy. The main VLM writes one Python cell per step. That cell is AST-checked and executed in the stateful kernel. Then stdout, newly created variables and images emitted through show() flow back to the model as the next observation. The loop repeats until the agent commits with ReturnAnswer(...). Two properties follow from this design. First, state accumulates: variables defined in an earlier cell remain available, so the agent can segment once and reuse the mask. Second, the observation channel is mixed, carrying both text and rendered images, which is what lets the agent inspect intermediate visual evidence before deciding. The AST check sits between generation and execution, and the README does not specify what it rejects, so treat it as a guardrail of unstated strictness. The kernel is the stateful component; the agent process manages Jupyter kernels, and the README notes these kernels are one of three independent services.

Three services, JSON registries and SLURM chain jobs

The runtime is split into a vLLM backbone, a GPU perception-tool server (Reconstruct and SAM3), and the agent with its Jupyter kernels. They coordinate through shared JSON registries. The README states that vLLM auto-discovery and load balancing are included, and that SLURM launch managers were used to reproduce every experiment in the paper, with auto-restarting chain jobs to survive job-time limits. That last detail tells you the intended deployment is a cluster where a single job cannot run long enough to finish an evaluation sweep. The README also states that each service is a plain Python entry point runnable on any GPU machine if you have no SLURM cluster, which is the path most readers will take. Note the coupling: the tool server and the backbone are separate processes with separate environments, so a failure in either surfaces as an agent that stalls rather than crashes.

Getting it running: the three commands in the README

The Quickstart gives a clone with submodules, a setup script and an experiment invocation. Cloning must be recursive because third-party perception code lives in submodules. The setup script is spatial_agent/scripts/setup.sh and the README estimates 15 to 30 minutes for the agent and vLLM environments. Configuration starts from a template: cp .env.example .env, then edit. API keys are optional if you self-host vLLM, which matters because the example model config in the Quickstart is gemini-3-pro.json, a hosted model. The run command is python -m spatial_agent.entrypoints.run with three flags: --dataset pointing at spatial_agent/config/dataset/erqa.json, --model pointing at spatial_agent/config/model/gemini-3-pro.json, and --concurrency 4. Datasets and models are therefore selected by JSON config file rather than by CLI arguments, which is what makes the same-hyperparameters claim across 20 benchmarks mechanically plausible. The README flags two prerequisites before SLURM runs: pre-downloading model weights is mandatory, and the vLLM and SLURM setup has extra steps documented in docs/installation.md and docs/running.md.

Where the design costs you: kernel state, safety checking and no releases

The persistent kernel is the source of the flexibility and the source of the failure modes. Because state carries across cells, an agent that overwrites a variable or leaves a mask in a stale coordinate frame can produce a plausible answer from corrupted intermediate evidence, and nothing in the described loop resets the kernel between samples. The AST safety check is the only barrier named, and the README does not document its policy, so you cannot reason about what generated code is allowed to do before you run it. Training-free is a real constraint as well as a selling point: there is no fine-tuned checkpoint to fall back on, so accuracy tracks the backbone, and the reported spread runs from 26B to 397B parameters. Operationally, the repository lists no releases, so you are tracking the main branch. The licence field is NOASSERTION, which means GitHub could not map the repository to a recognised licence; that is a fact about the metadata, not a statement about your rights, and it is the first thing to resolve with the maintainers if you plan to build on the code.

How this differs from a single-pass code agent

The nearest comparison the paper draws is a spatial agent that executes code in a single pass: it commits to a full analysis strategy before observing any intermediate result. SpatialClaw's difference is step granularity. One cell per step, conditioned on all prior outputs, means the agent can look at a segmentation mask or a depth rendering and then change course, for example switching from a top-down estimate to a multi-view reconstruction when the first attempt looks wrong. A structured tool-call interface sits on the other side of the trade: it is easier to validate and constrain because each call has a schema, but the abstract argues it offers less freedom to compose operations or tailor the analysis per task. SpatialClaw takes the permissive end of that axis deliberately. The cost is that correctness now depends on generated code being both safe and numerically sensible, which a schema-checked tool call would have prevented by construction.

What the 59.9% number does and does not tell you

The headline result is an average over 20 benchmarks spanning static and dynamic 3D and 4D spatial reasoning tasks, with the same system prompt, tool set and hyperparameters across all of them and across six VLM backbones from two model families (Qwen3.5/3.6 and Gemma4). An average over 20 heterogeneous benchmarks hides the per-benchmark distribution, and the repository's radar figure in assets/radar.png is normalised so that SpatialClaw forms the outer reference ring, which makes it a comparison aid rather than a raw accuracy chart. The README does not list per-benchmark scores in the text, and no releases were retrieved, so the version of the code that produced the published numbers is the main branch at the time of writing. If you intend to cite or extend the result, the paper PDF linked from the project page is the authoritative source for per-benchmark figures, and the SLURM managers in the repository are the path to reproducing them on your own hardware.

Maintenance surface and what to verify before adopting

The maintenance burden is the sum of three moving parts: the vLLM environment, the GPU perception-tool server with its SAM3 and Depth-Anything-3 submodules, and the agent environment with its Jupyter kernels. The setup script builds two conda environments, and pre-downloading weights is mandatory before any SLURM run, so a fresh machine is a multi-step provisioning job rather than a pip install. Because the submodules are third-party perception code, their licences travel with them and are separate from the repository's own undeclared licence. Concretely, verify three things in order: the licence situation for the repository and each submodule, whether docs/installation.md covers the GPU you have, and whether docs/running.md supports your execution model (plain Python entry points versus SLURM chain jobs). The last push recorded for the repository is 2026-07-18, so the code is active, but with no tagged releases you should pin a commit hash if you need a stable reference for a paper or a comparison.

Editorial conclusion

Adopt SpatialClaw if you are researching agentic spatial reasoning and want a reproducible harness: the same system prompt, tool set and hyperparameters are used across all 20 benchmarks and six VLM backbones, and the repository ships the SLURM managers used in the paper. Do not adopt it as a production perception service; the kernel, the GPU tool server and the vLLM backbone are three separately managed processes, and the licence is not declared. Before committing, read docs/installation.md and confirm the third-party submodule licences (SAM3, Depth-Anything-3) permit your use, and check whether the 59.9% average is computed on the benchmark split you actually care about.

Official sources

1. Issues
2. NVlabs/SpatialClaw on GitHub
3. Project website
4. README