LiteReality-Agent: turning a RoomPlan scan into a MuJoCo scene

What LiteReality-Agent does with a room scan

The input is not a generic point cloud. It is a walkthrough captured with the LiteReality Scanner app, which the README says records RGB frames, depth and the RoomPlan room.usdz file that the pipeline needs. The output is a simulation-ready environment: a reconstructed room whose objects are rigid bodies with mass, colliders and joints, exported to both URDF and MJCF. The README describes the whole flow as scan, then agent reconstruction, then MuJoCo.

The intended user is someone preparing robotics simulation scenes from real rooms, not someone building a game level. The 2026-09-12 release note states that every reconstructed room now loads as a MuJoCo scene, and the simulate stage writes realism_authoring/mujoco/scene.xml. That file, not a rendered image, is the deliverable. If you only need a textured mesh for visualisation, most of this pipeline is work you will not use.

Scene init is deterministic, authoring is agentic

A run splits into two stages, and the split matters because only one of them is reproducible in the ordinary sense. Scene init converts the capture into a seed room. The README calls it deterministic and notes it takes the capture and writes into run/<scan>/. Authoring then takes that seed room and an agent edits it until it matches the capture, comparing the seed against the original scan. The README says either half can be run on its own, which is why --through seed exists.

Around that core sit the layout and physics steps. The 2026-09-07 note describes layout agents settling the noisy layout of a raw scan into a collision-free layout before anything is built from it. The 2026-09-09 note says articulated objects carry their own mass, inertia, colliders and joints, following Articraft, and export to URDF and MJCF. Two optional passes sit on top of authoring: --polish adds object refinement and materials, and --quality-pass is a separate model-driven pass. The README explains the separation directly: the quality pass is the longest agent pass on a run and nothing downstream reads its output, so it is not on by default. That is a sensible default, and it also tells you the quality pass is a diagnostic, not a build step.

Installing LiteReality-Agent and running a first scan

The README recommends the Modal path for the heavy models, so an Apple Silicon Mac with no GPU is enough and detection fans out across several containers. Install the environment with uv, then copy the example environment file.

uv sync --frozen --extra modal --group dev
cp .env.example .env

Fill in .env. The README's table lists OPENAI_API_KEY for reference image generation, MODAL_TOKEN_ID and MODAL_TOKEN_SECRET from a free Modal account, BLENDER_PATH as the Blender install directory rather than the binary, and LR_SCANS_DIR as the folder holding your scans. The example block in the README looks like this.

OPENAI_API_KEY=sk-...
MODAL_TOKEN_ID=ak-...
MODAL_TOKEN_SECRET=as-...
BLENDER_PATH=/Applications/Blender.app/Contents/MacOS
LR_SCANS_DIR=~/scans

Deploy the models once per workspace, then run the sanity check the README presents as the readiness gate.

uv run litereality setup
SANITY_DEEP=1 uv run python sanity.py

If you have no scan yet, the README points at the example scans repository. Clone it and run a room end to end.

git clone https://github.com/LiteReality/example-scans.git
uv run litereality run example-scans/<scan>

The README states that the installed CLI is the only supported pipeline entry point. To see the build as it happens, authoring accepts --live, which starts a viewer before the room exists and waits for it; the README notes it prints its url again once the first build lands. To get the simulation scene, run the simulate stage, which writes realism_authoring/mujoco/scene.xml, and add --shake to measure what actually moves.

The GPU choice is the one real setup decision

TRELLIS and GroundingDINO need a GPU, and the README is explicit that this is the one real choice in the setup. The default is Modal, which the README calls recommended: nothing heavy runs on your machine, no local GPU is needed, and the free tier is described as covering the workload comfortably. The alternative is a Linux machine with a 24 GB or larger NVIDIA GPU, documented in deploy/local-gpu.md, where you point TRELLIS_PYTHON and optionally GROUNDING_DINO_PYTHON at local environments.

The cost structure is worth reading closely rather than skimming. Reference image generation is billed per scene through the image API and the README puts it typically under $1 per scene, while Modal's free tier is said to cover the model work. That combination is what makes the hosted path attractive for occasional use. A team reconstructing rooms continuously will want to compare that against the price of a 24 GB card, and the README does not offer a break-even figure.

Two dependencies are deliberately not optional, and the pyproject comments explain why. python-fcl backs a true-mesh collision check, and the comment states that a missing FCL turned the deterministic clash gate into a silent no-op because publish only records a warning when room_qc.correct exits non-zero. coacd has the same shape of failure with a worse outcome: without it, every concave link falls back to its own convex hull and the gate still reports a clean pass. Both are prebuilt wheels, so the cost is a few megabytes rather than a compiler. That reasoning is unusually candid for a project at this stage, and it tells you the authors have been bitten by a gate that quietly did nothing.

Where LiteReality-Agent is the wrong tool

The pipeline starts from a LiteReality Scanner capture. There is no documented path from an arbitrary mesh, a photogrammetry export or a CAD file, so if your rooms were not captured with that app you are outside the supported input. The README does not describe a converter for other capture formats.

The second constraint is the environment. Blender 5.x is required, tested on 5.1, and BLENDER_PATH must point at the install directory rather than the binary, which is an easy mistake to make and produces a failure that has nothing to do with your scan. An agent CLI has to be on your PATH, with claude as the default and codex supported through LR_AGENT_PROVIDER in models.env. That means the reasoning half of the pipeline depends on a tool you install and log into separately.

The third is maturity. The package version is 0.0.1 and the classifier says Development Status :: 3 - Alpha. The README advertises a technical report as coming soon, so there is no peer-reviewed description of the reconstruction quality to check against. Nothing in the README states accuracy figures, reconstruction success rates or how the agent behaves on a scan that is missing a wall. Treat a first run as an experiment, not a production step.

How this differs from Articraft and OpenIns3D

The README names Articraft directly, stating that articulated objects carry their own mass, inertia, colliders and joints following Articraft, and export to URDF and MJCF. That is an acknowledgement of shared ground on articulated asset generation. The difference is scope: LiteReality-Agent wraps that style of asset generation in a full pipeline that starts from a phone capture and ends at a room-level MuJoCo scene, with layout agents settling collisions before anything is built. Articraft is a component reference here, not a substitute for the capture-to-simulation path.

OpenIns3D is a different kind of comparison, and the honest position is that the README does not describe it, so any claim about how the two differ would be invented. What can be said is structural: LiteReality-Agent is built around RoomPlan output from a specific iOS app and around exporting physics-ready URDF and MJCF, so it competes on the simulation handoff rather than on open-vocabulary 3D instance detection as a standalone task.

If your real requirement is a simulation-ready room from a real capture, the alternative that matters most is simply doing the assembly yourself: run an image-to-3D model, place the assets, and hand-write the MJCF. That path gives you full control and no dependency on an agent CLI, at the cost of the layout and collision work the layout agents are meant to absorb.

Licence, maintenance and the cost of upgrading

The repository is Apache-2.0, declared both in the package metadata and as a LICENSE file at the top level. That is a permissive licence with an explicit patent grant, and it is the same identifier the classifier records. It does not resolve the terms of the models the pipeline calls: TRELLIS, GroundingDINO and DINOv2 are separate projects with their own licences, and the README does not state which terms apply to their weights or to outputs generated through Modal. Check those separately before shipping generated assets; this is a factual gap in the documentation, not a legal opinion.

The repository is not archived, and the last push was on 2026-09-11, so the codebase is being changed. That is not the same as a stable interface. At version 0.0.1 with an alpha classifier and no retrieved releases, expect the CLI surface to move. The README already shows the shape of that churn: the simulate stage and the whole simulation-ready release arrived within days of each other in September 2026. Pin your uv.lock, keep your scans, and re-run sanity.py after any upgrade rather than assuming a passing workspace still passes.

The upgrade cost that is easy to miss is the agent CLI. Because authoring is driven by an external claude or codex process, a change in that tool can change your reconstruction without any change in this repository's version number.

Editorial conclusion

Adopt LiteReality-Agent if you already capture rooms with the LiteReality Scanner and want the result as bodies with mass, colliders and joints rather than a baked mesh, and if you are comfortable with Blender 5.x, an image API key and a Modal account or a Linux box with a 24 GB GPU. Do not adopt it if you have no RoomPlan capture, no GPU path, or you only need a static visual mesh, because the whole pipeline is built around a scan plus hosted or local TRELLIS and GroundingDINO, and the project is still 0.0.1 alpha. Verify first that sanity.py passes in your workspace, since the README presents it as the readiness check before any run.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. LiteReality/LiteReality-Agent on GitHub
4. Project website
5. README