Code-as-World: executable world representations discovered by agents

The problem Code-as-World attacks: pixels are evidence, not structure

The README states the premise directly: pixels are evidence of the physical world, not its ontology. A video records how a scene appeared from one viewpoint at one moment. It does not state what objects exist, how they are connected, or which equations govern their motion. Anyone who has tried to train a model to answer "how far will this ball travel" from raw frames knows the gap: the model can describe the frame and still fail the arithmetic.

Code-as-World's answer is to make the world representation executable. Instead of a latent vector or a caption, the target artifact is code that can be run and checked against observations. The README frames this as turning raw observations into reusable physical data: explicit states, dynamics and mechanisms. The audience is narrow and specific. This is for researchers working on physical reasoning, world models, video understanding and code generation, and for teams that need quantitative physical supervision rather than more captions. It is not aimed at application developers who want a physics engine with a stable API.

How the discovery loop and the released checkpoints fit together

The repository layout reflects two distinct pieces of work. The first is the agentic discovery process described in the technical report: an agent proposes code as a world representation, runs it, and verifies the result against observations, iterating until the representation holds up. The README states that this process turns abundant observations into reusable physical data covering states, dynamics and mechanisms.

What is actually shipped in the repository is the second piece: the models trained on that data, plus the tooling to run them. The top-level entries are code_as_world/, examples/, requirements/, assets/, the paper PDF and the licence. The code_as_world package exposes three entry points that the README documents as modules: code_as_world.evaluation for QuantiPhy scoring, code_as_world.simulation for the MuJoCo ballistic soccer case, and code_as_world/templates/qwen3_5_no_think.jinja for the vLLM chat template. The checkpoints, Code-as-World-VL-4B and Code-as-World-VL-9B, live on Hugging Face under MirroS-Lab, not in the git repository.

That split matters when you plan an evaluation. Cloning the repository gets you the harness and the example, not the weights. The discovery loop that the paper describes is documented in the technical report and on the project page; the README does not present a standalone command that runs the full agentic discovery pipeline end to end.

Installing Code-as-World and running a first evaluation

The README asks for Python 3.10 or 3.11 on a CUDA host. Start with the clone and the inference requirements, then pull the weights you intend to use. The hf download commands place the two checkpoints under weights/4b and weights/9b inside the repository.

git clone https://github.com/MirroS-Lab/Code-as-World.git
cd Code-as-World
python -m venv .venv
source .venv/bin/activate
pip install -r requirements/inference.txt
hf download MirroS-Lab/Code-as-World-VL-4B --local-dir weights/4b
hf download MirroS-Lab/Code-as-World-VL-9B --local-dir weights/9b

The evaluation depends on QuantiPhy, which is a separate repository and a separate dataset. Clone it and fetch its validation videos before running anything.

git clone https://github.com/Paulineli/QuantiPhy.git /path/to/QuantiPhy
hf download PaulineLi/QuantiPhy-validation \
  --repo-type dataset \
  --local-dir /path/to/QuantiPhy-validation

With both in place, the evaluation module takes the model size as its first positional argument. A 4B run looks like this.

python -m code_as_world.evaluation 4b \
  --input-csv /path/to/QuantiPhy/quantiphy_validation.csv \
  --video-dir /path/to/QuantiPhy-validation/validation_videos

Swap 4b for 9b for the larger checkpoint. According to the README, each run writes an evaluator-compatible prediction CSV, a single-run metric summary and the raw generations into outputs/quantiphy/. To score those outputs with QuantiPhy's own evaluator, install pandas and point the evaluator at the output directory.

pip install pandas
python /path/to/QuantiPhy/evaluator.py \
  outputs/quantiphy \
  outputs/quantiphy_metrics \
  --gt_file /path/to/QuantiPhy/quantiphy_validation.csv

If you would rather call the model as a service, the README documents a vLLM recipe with the bundled chat template, a max-model-len of 4608, a GPU memory utilization of 0.90 and video decoding at 16 frames. Note the template kwargs disable thinking mode, and the media IO kwargs specify the openpangu video backend. Those flags are specific to this checkpoint family; do not assume they transfer to another Qwen-based model.

The MuJoCo example is a demonstration, not the discovery pipeline

The repository ships one runnable physical example under examples/ballistic_soccer/. The README describes it as a video-driven abstraction example and gives two commands. Installing requirements/simulation.txt pulls in MuJoCo, and running the simulation module renders the case and writes both a video and a trajectory into outputs/simulations/.

pip install -r requirements/simulation.txt
python -m code_as_world.simulation

A --no-render flag is documented for the case where only the trajectory is needed. This is the closest thing in the repository to a hands-on look at what an executable world representation means in practice: a scene, a trajectory, and a rendered result you can inspect. Treat it as a worked example rather than as the agentic loop. The README does not describe a CLI that takes an arbitrary video and returns discovered code; the discovery process is described in the technical report, and the repository's shipped entry points are evaluation and simulation.

Where Code-as-World is the wrong tool

The most obvious limitation is scope. Two checkpoints, one benchmark (QuantiPhy), one simulation example. If your problem is general video understanding, object tracking or scene reconstruction, the acknowledgements table points at the components the project itself builds on: SAM 3, DA3, VGGT-Omega and SAM 3D for perception and reconstruction, Wan and VACE for video generation. Those are the tools for that layer, and Code-as-World assumes you already have the observations.

Hardware is the second constraint. The README requires a CUDA host and Python 3.10 or 3.11. The vLLM recipe sets a max-model-len of 4608 and a GPU memory utilization of 0.90, which tells you the serving path is designed for a single GPU with substantial memory rather than for CPU inference or a laptop. There is no documented CPU fallback and no quantized variant.

The third limitation is operational. The README does not document rollback, versioned checkpoint compatibility, or an upgrade path between releases. There are no retrieved releases at all, so the checkpoint tags are the version boundary you have. If you need a dependency with a stability contract, this is not it. It is also the wrong choice if your evaluation needs to be reproducible on a machine without network access, because both the weights and the QuantiPhy validation videos are fetched with hf download from Hugging Face.

How it differs from a simulator or a physics engine

The natural comparison is MuJoCo, which this project already depends on for its example. A simulator like MuJoCo executes a model that a human wrote: you specify bodies, joints and contacts, and the engine integrates them. Code-as-World inverts the direction of authorship. The representation is the thing being discovered, and MuJoCo is one of the execution backends the project uses to check whether a candidate representation behaves like the observed world.

A second comparison is against latent world models, the family the related searches call world models and JEPA. Those learn a predictive state space and reason inside it; the state is not readable as code. Code-as-World's bet, as the README puts it, is that a world becomes intelligible when it can be represented, executed and verified. Executability is the difference: you can read the artifact, run it, and compare its output against the video it came from. The cost of that bet is that the discovered representation is only as good as the verification step, and the README does not describe what happens when a candidate representation passes verification on the sampled frames but fails outside them.

Licence, maintenance and what an upgrade costs

The repository is Apache-2.0. That covers the code in this repository, including the evaluation module, the simulation example and the chat template. It does not automatically cover the model weights, which are distributed separately on Hugging Face, or the QuantiPhy repository and dataset, which have their own terms. If you plan to redistribute generated world code or derivatives of the checkpoints, check each of those sources rather than assuming the Apache-2.0 grant travels with them. None of this is legal advice.

On maintenance: the repository is not archived, and the last push was on 2026-08-31. The news section records the technical report and the project page going live on 2026/08/27, and the TODO list shows the checkpoint release and the report release both checked off. That is a release-shaped repository, not a long-running project with a deprecation policy. Upgrading means re-downloading weights and re-running the QuantiPhy evaluation, because the README documents no migration notes and no changelog. Budget the cost of re-validating your own numbers against the new checkpoint, not just the download.

Editorial conclusion

Adopt Code-as-World if you are doing research on physical reasoning, world models or code generation, and you want a released checkpoint plus a QuantiPhy evaluation path you can rerun. Do not adopt it as a production perception stack or as a general-purpose simulator: the repository is a research release with two checkpoints, one evaluation harness and one MuJoCo example, and the README documents no service-level guarantees, no rollback and no upgrade path. Before committing, verify three things: that your CUDA host has enough GPU memory for the 9B checkpoint at the documented max-model-len of 4608, that the QuantiPhy validation CSV and video directory are the ones the evaluation module expects, and that the Apache-2.0 licence on this repository is compatible with how you plan to redistribute the generated world code.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. MirroS-Lab/Code-as-World on GitHub
4. Project website
5. README