# CaP-X measures robot coding agents across 39 tasks and eight benchmark tiers > CaP-X is an open-access research framework for Code-as-Policy agents, the pattern where a vision language model writes Python that composes perception and control primitives for a robot. It ships a gym of simulator tasks, a benchmark split into single-turn and multi-turn tiers, a training-free agent, and a GRPO reinforcement learning loop, and it needs a CUDA GPU, Python 3.10 and a simulator-specific environment before it will run a single trial. **capgym/cap-x** — [ICML 2026] A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation - Repository: https://github.com/capgym/cap-x - Website: https://capgym.github.io/ - Stars: 882 · Forks: 132 - Language: Python - License: MIT - Published: 2026-10-20 · Updated: 2026-10-20 · Language: en - Canonical page: https://hysenlabs.com/projects/capgym-cap-x ## Code-as-Policy means the model writes the robot's code The framework is built around one idea, and the README names it in the first line of the overview. CaP-X is an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. A Code-as-Policy agent is one where the model does not emit a low-level action. It writes Python code that composes perception and control primitives, and that program is what drives the robot. The consequences of that choice run through the whole framework. The agent needs access to a perception stack, since the code it writes calls into segmentation, grasp planning or inverse kinematics services. It needs an environment that can execute the code. And evaluating it means running a program, not classifying an answer, which is why the harness is a robotics simulator setup rather than a question answering one. The paper is on arXiv, the project page is at capgym.github.io, and the author list spans NVIDIA, UC Berkeley, Stanford University and Carnegie Mellon University, with a note that the first five authors contributed equally and that two others share equal advising. The project is deliberately research facing. The repository carries a `web-ui/` directory, a `docs/` directory with per-simulator task guides, a `tests/` directory and a `verl_agent_reward/` directory, and the only tagged release is CaP-X v0.0.1 from 2026-04-01, with the last push on 2026-05-28. The licence is MIT, with the manifest stating it in text. The package metadata gives the compact definition of the scope, describing CaP-X as Code-as-Policies eXtended with RL for V(L)A code-as-action with simulators, and requiring Python from 3.10 up to but not including 3.13. ## Four components that share one simulator stack The README breaks the framework into four pieces, and it is worth reading them as a pipeline rather than as a feature list. CaP-Gym is the environment layer, described as interactive Gymnasium environments where agents control robots by generating Python code that composes perception and control primitives, with 39 tasks across Robosuite, LIBERO-PRO and BEHAVIOR. CaP-Bench is the measurement layer, a systematic benchmark evaluating coding agents across abstraction levels, interaction modes and visual grounding modalities, split into 8 tiers, S1 to S4 for single-turn and M1 to M4 for multi-turn. The tiering is the interesting design choice. A single-turn tier means the agent writes one program and the program runs to completion, while a multi-turn tier means the agent can observe the result of its own code and revise it, which is a different capability and a different failure mode. CaP-Agent0 is the baseline agent, a training-free agentic framework with multi-turn visual differencing, auto-synthesized skill libraries and parallel ensembled reasoning. Training-free matters for a benchmark, because a baseline that required training would confound the comparison with the training budget rather than the method. CaP-RL closes the loop, applying reinforcement learning on the coding agent via GRPO, using environment rewards to post-train language models, and the README states that it transfers from simulation to real with a minimal gap. The dependency list corroborates the architecture. Gymnasium is the environment interface, `torch` and `torchvision` are the model stack, `ray` is pinned to an exact version for distributed execution, `openai` and `transformers` are the model clients, and `sam3` is a direct dependency, which tells you segmentation is part of the default environment rather than an optional extra. ## One simulator at a time, because the Robosuite versions clash The installation section has a warning that most robotics frameworks do not need to state, and it is the first thing that will cost you time. CaP-X uses uv for dependency management and requires Python 3.10 and a CUDA capable GPU. The base install clones the repository with submodules, because the simulator dependencies are pulled in as git submodules. ```bash git clone --recurse-submodules https://github.com/capgym/cap-x && cd cap-x ``` Then the environment is created and synced. ```bash uv python install 3.10 && uv venv -p 3.10 ``` ```bash uv sync ``` The conflict is between simulator families. The README says to pick one simulator family to install, because Robosuite 1.5.0 and LIBERO, which needs `robosuite==1.4.0`, would be in conflict. Robosuite alone is a single extra. ```bash uv sync --extra robosuite ``` LIBERO is worse, because it requires a separate virtual environment with a different Python version. ```bash uv venv .venv-libero --python 3.12 source .venv-libero/bin/activate uv sync --active --extra libero --extra contactgraspnet ``` BEHAVIOR is the heaviest of the three. It runs on NVIDIA Isaac Sim through OmniGibson, needs Python 3.10 and CUDA 12.x, and its own installer script brings in OmniGibson, Isaac Sim, BDDL and cuRobo and downloads robot and scene assets. The README then describes a post-install fix that copies cuRobo JIT headers into the installed package, and a fix for headless servers where a duplicate Vulkan ICD file has to be removed because it causes a segfault on multi-GPU systems. There is a 130+ task documentation file for LIBERO, and the BEHAVIOR `--dataset` flag also downloads 2025 challenge task instances. ## Perception servers are launched from the YAML config The first quick start step is the one that most distinguishes this framework from a scripted evaluation. Perception servers, namely SAM3, ContactGraspNet and PyRoKi, are auto-launched by the YAML config when you run an evaluation, and the README says no manual setup is required for most configs. That is the mechanism that makes Code-as-Policy evaluation practical. The code the agent writes calls into a segmentation model, a grasp planner and an inverse kinematics solver, and all three have to be running as services before the evaluation starts. You can pre-launch them yourself, which the README suggests when you want to share the servers across several evaluation runs, and the launcher takes a profile and can print the allocation without starting anything. ```bash uv run --no-sync --active capx/serving/launch_servers.py --profile default ``` The profiles are listed in the README: `default` starts SAM3, GraspNet and PyRoKi with automatic GPU allocation, `full` adds all perception servers including OWL-ViT and SAM2, and `minimal` starts PyRoKi alone, which the README notes is for oracle or privileged evaluations. Automatic GPU allocation is the detail that makes this tractable. Perception models and the policy model want the same device, and the launcher resolves that. The `--dry-run` flag is there to preview the allocation before committing. One external dependency needs credentials. SAM3 weights require HuggingFace access, so you request access on the SAM3 repository and authenticate with `huggingface-cli login`, and the weights are then cached after the first download. The optional extras cover the rest of the stack: VeRL and GRPO for RL training, ContactGraspNet for grasp planning, and cuRobo for GPU accelerated inverse kinematics and motion planning. ## Models are reached through a local OpenAI-compatible proxy The evaluation harness does not talk to a model provider directly. It queries an LLM through a local proxy that exposes an OpenAI-compatible API, and the README shows OpenRouter as the example. ```bash echo "sk-or-v1-your-key-here" > .openrouterkey ``` ```bash uv run --no-sync --active capx/serving/openrouter_server.py --key-file .openrouterkey --port 8110 ``` Two details are worth noting. The key is written to a file rather than passed as an argument, and the README says `.openrouterkey` is git-ignored, which is the right way to keep a credential out of shell history and out of a process listing. And the default server URL in the configs is `http://127.0.0.1:8110/chat/completions`, so the harness is already pointed at that port and endpoint. The proxy is not locked to one provider. The README points to `docs/configuration.md` for all provider options, which it lists as OpenRouter, vLLM and custom. That matters for a research group, because evaluating against a hosted frontier model and against a locally served open model are different experiments with different cost and reproducibility profiles, and the proxy makes both reachable through the same interface. The OpenAI-compatible shape of the API is also why the dependency list includes the `openai` package. Whatever model sits behind the proxy, the harness speaks the same protocol to it. ## Running a trial is a config path and a model name The evaluation entry point is a launcher script, and the arguments are minimal. ```bash uv run --no-sync --active capx/envs/launch.py \ --config-path env_configs/cube_stack/franka_robosuite_cube_stack.yaml \ --model "google/gemini-3.1-pro-preview" ``` The configuration file selects the environment and the task, the `env_configs/` tree holding one YAML per task with names that encode the robot, the simulator and the task. The model is given as a provider and model name in one string, which fits the OpenAI-compatible proxy, since the proxy is the thing that resolves that name. The comment on that command in the README describes it as a Robosuite single-turn benchmark with 100 trials and 12 parallel workers, and the multi-turn variant is the same invocation with a different config file, one whose name ends in `_multiturn_vdm` for the visual differencing mode. ```bash uv run --no-sync --active capx/envs/launch.py \ --config-path env_configs/cube_stack/franka_robosuite_cube_stack_multiturn_vdm.yaml \ --model "google/gemini-3.1-pro-preview" ``` LIBERO runs through the same launcher once its separate environment is active, and BEHAVIOR additionally requires the b1k virtual environment, with the Isaac Sim end user licence acceptance set as an environment variable on the command. So the sequence for a first evaluation is: install one simulator, start the perception servers, start the LLM proxy, then point the launcher at a config and a model. Four steps, each with a documented failure mode, which is roughly the amount of setup a robotics evaluation deserves. ## What it costs to run and what it costs to maintain The resource requirements are the first filter for anyone evaluating this framework. A CUDA capable GPU is required, not optional, and the perception server launcher has to allocate device memory between SAM3, the grasp planner, the kinematics solver and the policy model. BEHAVIOR adds Isaac Sim, a dataset download of robot and scene assets, and a licence acceptance step. The optional `molmo` extra pins `torch` to an exact version and pulls in vLLM, which is the path for serving a policy model locally rather than through a hosted provider. The dependency surface is large and deliberately pinned in places. `ray` is pinned to an exact version, the robosuite version differs between the two simulator families, and the cuRobo JIT header copy is a manual post-install step. None of that is unusual for a research framework that has to reproduce results, but it does mean the environment is not a one-line install and that a rebuild of the machine is a real cost. On the maintenance side, the facts are thin. The repository is not archived, the last push was on 2026-05-28, and the only tagged release is CaP-X v0.0.1 from 2026-04-01. The manifest version is 0.1.0 while the release tag is v0.0.1, so the project has not settled on a version scheme. There is a `uv.lock` file, which pins the Python dependency tree, and the `docs/` directory holds the task guides and the configuration reference, so the documentation for a rebuild exists in the repository. The realistic picture is a research artifact maintained alongside the paper, with a working harness that a group can adopt for a comparison but that they should expect to maintain themselves, especially across simulator or model version changes. ## Conclusion CaP-X fits a research group that wants a reproducible harness for comparing code-as-policy agents across simulators, with the perception stack, the LLM proxy and the evaluation loop already wired together. Skip it if you do not have a CUDA capable GPU and the patience to install Isaac Sim, or if a single policy rollout is what you need. Verify first which simulator you will use, since the README warns that the Robosuite and LIBERO versions conflict and LIBERO needs its own virtual environment, and confirm that the model you intend to query is reachable through the proxy the harness expects. ## FAQ ### What is CaP-X and what does it benchmark? CaP-X is an open-access framework for studying Code-as-Policy agents in robot manipulation, where the agent writes Python code that composes perception and control primitives. Its CaP-Bench component evaluates coding agents across abstraction levels, interaction modes and visual grounding modalities, in 8 tiers, S1 to S4 for single-turn and M1 to M4 for multi-turn. ### How do I install CaP-X? The project uses uv, requires Python 3.10 and a CUDA capable GPU, and is installed by cloning with `git clone --recurse-submodules https://github.com/capgym/cap-x && cd cap-x`, creating the environment with `uv python install 3.10 && uv venv -p 3.10`, and running `uv sync`. You then add the extra for the single simulator family you intend to use. ### Why do I need separate environments for Robosuite and LIBERO? The README says to pick one simulator family, because Robosuite 1.5.0 and LIBERO with `robosuite==1.4.0` would be in conflict. LIBERO requires a separate virtual environment, created with `uv venv .venv-libero --python 3.12`, while BEHAVIOR runs on Isaac Sim through OmniGibson with its own installer. ### How does the CaP-X evaluation harness call a language model? Through a local proxy that exposes an OpenAI-compatible API, started for example with `uv run --no-sync --active capx/serving/openrouter_server.py --key-file .openrouterkey --port 8110`. The default server URL in the configs is `http://127.0.0.1:8110/chat/completions`, and the documentation lists OpenRouter, vLLM and custom providers. ### Do I need to start the perception servers manually in CaP-X? No. The README says the perception servers, SAM3, ContactGraspNet and PyRoKi, are auto-launched by the YAML config when you run an evaluation, with no manual setup required for most configs. You can pre-launch them with `capx/serving/launch_servers.py --profile default` if you want to share them across runs, and `--dry-run` previews the GPU allocation. ## Sources - [capgym/cap-x on GitHub](https://github.com/capgym/cap-x) - [License: MIT](https://github.com/capgym/cap-x/blob/main/LICENSE) - [Project website](https://capgym.github.io/) - [README](https://github.com/capgym/cap-x/blob/main/README.md) - [Releases](https://github.com/capgym/cap-x/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/capgym-cap-x