# SceneSmith: generating simulation-ready indoor scenes from a text prompt

> The code behind an ICML 2026 Spotlight paper, turning a written description into a 3D scene with separable objects and estimated physical properties, ready to drop into a physics simulator. The pipeline leans on SAM3D for asset generation, Blender for rendering and an LLM agent for the scene logic, and the setup needs real GPU memory.

**nepfaff/scenesmith** — Code for "SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes", ICML 2026 Spotlight

- Repository: https://github.com/nepfaff/scenesmith
- Website: https://scenesmith.github.io/
- Stars: 639 · Forks: 59
- Language: Python
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/nepfaff-scenesmith

## What the paper claims, and what the code does

SceneSmith is the code for the paper SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes, published at ICML 2026 as a Spotlight, with an arXiv identifier of 2602.09153 and a project website at scenesmith.github.io. The authors are Nicholas Pfaff and Thomas Cohn at the Massachusetts Institute of Technology, with Sergey Zakharov and Rick Cory at the Toyota Research Institute, and Russ Tedrake also at MIT.

The headline claim in the readme is fully automated text-to-scene generation. A whole community center was generated by SceneSmith without any human intervention, from a single 151-word text prompt. Beyond explicitly specified elements, SceneSmith places additional objects from inferred contextual information, such as ping pong paddles and balls placed near a ping pong table.

Two properties of the output are what make it simulation-ready rather than merely visual. Objects are generated on-demand and are fully separable, meaning not composite, and they include estimated physical properties, which is what enables direct interaction within a simulation. The resulting scenes are immediately usable in arbitrary physics simulators, with robots added for demonstration.

The point of separability is worth pausing on, because it is the hard part. A scene that a simulator can act on needs to know which object is which, and it needs to know things like how heavy each one is or whether it can be knocked over. A renderer that outputs a single mesh with everything welded together is much easier to produce and much less useful. So the design goal is not a picture of a room, it is a room made of things.

The licence is MIT and the repository is not archived, with the last push on 2026-06-20.

## The asset generation backend is where the GPU requirement comes from

SceneSmith supports two backends for open-set 3D asset generation, and the choice is really a choice about how much GPU memory you have and how much quality you need.

The Hunyuan3D-2 backend gives lower-quality asset generation but fits into 24GB GPU memory, and the results will be significantly worse than when using SAM3D. The readme recommends it only as a proof-of-concept if you are highly GPU memory limited, and says to use SAM3D for production results. SAM3D gives higher-quality asset generation but requires 32GB GPU memory, and it is the backend used in the paper.

Setting up SAM3D takes a few steps. You request access to the SAM models on HuggingFace, at the SAM-3D and SAM3 pages, and automatic approval usually happens within 30 minutes. You authenticate with the CLI, then run the install script, which clones the SAM3 and SAM 3D Objects repositories, installs dependencies including pytorch3d, gsplat, kaolin and nvdiffrast, and downloads model checkpoints of about 5GB. Then you enable it in your agent config:

```yaml
# In your agent config files
asset_manager:
  backend: "sam3d"
```

There is a third path that needs no generation backend at all. If you are using strategy: "hssd", which is retrieval from the HSSD library, no generation backend is needed.

That third path is the one to understand before installing anything. Retrieval from a prepared library is a fundamentally different operation from generating an asset on demand, and the readme's emphasis on the install steps for the two generation backends, with a warning that installation is not complete yet, suggests how much of the setup effort sits in the generation path.

The repository tree backs this up with an external directory for the cloned model repositories and checkpoints, a data directory for the datasets, and an install_hunyuan3d.sh and install_sam3d.sh in the scripts directory.

## Installing the dependencies is the easy part, the data is not

The install section is a uv based flow, and it is short. The repository uses uv for dependency management, so you install uv, sync the dependencies into a .venv, and activate the environment.

```sh
curl -LsSf https://astral.sh/uv/install.sh | sh
```

```sh
uv sync
```

The sync command installs the dependencies including dev tools like pytest into .venv, and uv sync --no-dev installs without the dev dependencies. Then you source the environment and install the pre-commit hooks.

The readme is explicit that installation is not complete at this point. The default configuration requires additional data and model checkpoints, and it lists three things to continue with: the SAM3D backend for 3D asset generation as the default backend, ArtVIP for articulated objects, and AmbientCG for PBR materials.

Those three are the data setup sections of the documentation, and they map onto the data directory in the tree: the HSSD dataset, the Objaverse and Objaverse-THOR setup for objects, the articulated object datasets, and the AmbientCG PBR materials. The data directories carry the heavy lifting that the pip install does not.

So the realistic picture is that the Python environment is the cheap part and the data is the expensive part, both in disk and in time to fetch and prepare.

The dependency list in the project manifest tells you what the pipeline is actually made of. bpy 4.5.4 is the Blender Python API, and there is a pyvirtualdisplay, so the rendering is headless Blender rather than a separate renderer. drake 1.47.0, manifold3d, trimesh, shapely, rtree, coacd, xatlas, pyatlas-adjacent tooling, vhacdx, python-fcl, pymeshlab and mathutils are the geometry stack, for meshes, collision, decomposition and UV atlasing. openai-agents is the agent framework, with openai and google-genai for the models. hydra-core and omegaconf are the configuration system, which is what the backend setting in the YAML above belongs to. And there is bpy, open-clip-torch, diffusers and transformers for the generative side, plus wandb for experiment tracking and flask, which suggests a local service surface.

## Multi-GPU rendering, and why bubblewrap is a dependency

The pipeline parallelises Blender rendering across GPUs, and the mechanism for that is bubblewrap, which you install on Ubuntu or Debian with:

```sh
# Ubuntu/Debian
sudo apt-get install bubblewrap
```

This enables GPU isolation for EEVEE Next rendering, preventing out-of-memory errors when running many scenes in parallel. Each scene's BlenderServer is isolated to a single GPU via OS-level namespacing. Without bubblewrap, all Blender instances share GPU 0.

That last sentence is the whole problem in one line. A scene generation pipeline wants to render many scenes at once, and Blender's EEVEE Next renderer will happily allocate across every GPU it can see, so several workers will pile onto device 0 and exhaust its memory. Bubblewrap gives each worker a namespaced view of the hardware where only one GPU is visible, and that is what makes parallel rendering stable rather than a source of out-of-memory failures.

The project recommends a multi-GPU node for the best scene generation throughput when using generated assets, and reports that AWS g6e.48xlarge instances work well with the system.

Note what this costs you in a container. The Compose file adds SYS_ADMIN and SYS_CHROOT capabilities and sets apparmor=unconfined and seccomp=unconfined, because bubblewrap needs to create namespaces that a default container profile forbids. That is a meaningful reduction in isolation for the container, and it is the price of the GPU isolation the pipeline needs. It is a reasonable trade for a rendering worker, and it is not something to copy into a container that handles untrusted input.

## The pipeline has stages you can stop, and experiments you can branch from

The documentation's structure suggests a system with a state machine, and that is where the engineering interest is for anyone who wants to use this rather than just read the paper.

There is a Pipeline Stage Control section, a Branching section described as branching from a previous experiment, a Multi-GPU Support section under scene generation, and a Visualisation section, plus an Export section and a Robot Evaluation section. The top level entry point in the tree is main.py, and the package itself lives in the scenesmith directory.

Stage control matters because the expensive part is the asset generation and the rendering, not the scene reasoning. If a pipeline can be run up to the point where the objects are decided and then stopped, inspected, and resumed, you can iterate on the scene logic without paying for a full regeneration each time. The branching feature, described as branching from a previous experiment, is the same idea applied across runs.

The configuration system is Hydra with OmegaConf, and the configurations directory at the root of the tree holds the agent configuration files where the asset_manager backend setting lives. That combination, Hydra configs plus a stage-controllable pipeline, is a research project designed to be run many times with different settings, and the repository includes a tests directory with pytest as the dev dependency, along with pytest-asyncio in auto mode.

The robot evaluation section is what ties the whole thing back to the claim. A scene with estimated physical properties is only interesting if something can act on it, so the evaluation harness with robots is the check that the output is usable in a simulator rather than merely a picture.

## Where SceneSmith is the wrong tool

There are clear boundaries here, and they come from the ambition of the approach rather than from any defect in it.

The first is that it is a paper artefact. The repository description says it is the code for a specific paper, the documentation is organised around reproducing that work, and there are no GitHub releases. Nobody is promising a supported product with a compatibility policy. If you need something you can depend on for the next three years, this is not that, and the licence being MIT does not change the fact that you would be adopting research code.

The second is the compute. SAM3D needs 32GB of GPU memory, Hunyuan3D-2 needs 24GB, the checkpoints are about 5GB, and the datasets are additional. On top of that, the multi-GPU path requires specific container privileges. A team without a GPU machine of that class is not going to run the default backend, and the readme's own wording about the Hunyuan3D path being a proof-of-concept for memory limited setups is a warning about quality, not about feasibility.

The third is the dependency on gated models. The SAM models require you to request access on HuggingFace and wait, with automatic approval usually within 30 minutes, which means your build is not reproducible from a clean machine in a way you control.

The fourth is the input side. The demonstration is a text prompt, and the readme says the system places additional objects from inferred contextual information. That inference is the interesting part and also the unpredictable part, and for anything with a correctness requirement rather than a research requirement, a scene whose contents depend on a language model guessing is a hard dependency to reason about.

And the fifth is the Python version. requires-python is >=3.11, <3.12, with the toolchain configured for Python 3.11 in the Dockerfile, including the deadsnakes PPA and the alternative setup to make python3.11 the default. A narrow pin like that is reasonable for a research reproduction and a nuisance for anything else.

## Against building the scene by hand, and against a static asset library

Two comparisons make the value clearer.

The first is against authoring a scene by hand in Blender or a similar tool. The difference in approach is where the decisions come from. Hand authoring gives you total control and total effort, and the result is a scene whose contents you chose, so it is exactly what you need for a specific experiment. SceneSmith gives you a scene from a description, with the object set inferred rather than specified, which is the right shape when you need many varied scenes rather than one perfect one.

For a robotics researcher who needs a hundred cluttered indoor environments to test a grasping policy, the generated and varied kind is worth a lot, and the inferred objects are a feature rather than a hazard, because clutter is what you wanted. For a benchmark where the exact scene matters, the inference is a problem, and hand authoring wins.

The second is against a static asset library, which is what the HSSD strategy effectively is. A curated library of prebuilt 3D objects is simpler, faster and more predictable, because the assets are already there and already good. Generating them on demand is slower and needs a GPU, and in exchange it gives you objects that were not in any library, which is the point when you need open-set coverage.

So the choice between the two backends is really a choice between a fixed vocabulary of objects with predictable quality and an open vocabulary with a GPU bill. The readme's own recommendation tells you which one the authors think is better for production results, and the Hunyuan3D-2 option is framed as the fallback when memory is the binding constraint.

## Conclusion

Adopt SceneSmith if you need synthetic indoor scenes that a physics simulator can act on, with objects that are separable and carry estimated physical properties, rather than a rendered image or a mesh soup. Skip it if you need a light, dependency-light scene generator, since the default path needs gated model access, tens of gigabytes of checkpoints and a large GPU, and the project is a paper artefact rather than a product. Verify first which asset backend your hardware can hold, since SAM3D wants 32GB and Hunyuan3D-2 fits in 24GB with visibly worse results, and read the pipeline stage control and branching documentation before running anything large.

## FAQ

### What does SceneSmith generate?

It generates simulation-ready indoor 3D scenes from a text prompt with no human intervention, for example a community center from a 151-word prompt. Objects are generated on demand, are fully separable rather than composite, and include estimated physical properties so the scene can be used directly in a physics simulator.

### Which asset generation backends does SceneSmith support?

Two: SAM3D for higher quality requiring 32GB GPU memory, which is the backend used in the paper, and Hunyuan3D-2 for lower quality that fits into 24GB and is recommended only as a proof-of-concept when GPU memory is limited. A third option, strategy: "hssd", retrieves from the HSSD library and needs no generation backend.

### How do I install SceneSmith?

The repository uses uv. Install uv with curl -LsSf https://astral.sh/uv/install.sh | sh, run uv sync to install dependencies including dev tools into .venv, then source .venv/bin/activate and run pre-commit install. The readme stresses that installation is not complete until the data and model checkpoints are set up.

### What are the environment variables SceneSmith needs?

OPENAI_API_KEY is required for GPT-5 agents and default image generation. GOOGLE_API_KEY is optional and only needed for the Gemini image generation backend, and OPENAI_TRACING_KEY is an optional separate key so traces can land on a different account than the one used for billing.

### Why does SceneSmith need bubblewrap?

For GPU isolation during parallel EEVEE Next rendering. Each scene's BlenderServer is isolated to a single GPU through OS-level namespacing, which prevents out-of-memory errors when running many scenes at once. Without bubblewrap, all Blender instances share GPU 0. The Compose file adds SYS_ADMIN and SYS_CHROOT and disables apparmor and seccomp for this reason.

### What licence is SceneSmith released under and who wrote it?

MIT. The paper is SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes, an ICML 2026 Spotlight with arXiv identifier 2602.09153, authored by Nicholas Pfaff, Thomas Cohn, Sergey Zakharov, Rick Cory and Russ Tedrake, with MIT and the Toyota Research Institute. The repository is not archived and the last push was on 2026-06-20.

## Sources

- [Issues](https://github.com/nepfaff/scenesmith/issues)
- [License: MIT](https://github.com/nepfaff/scenesmith/blob/main/LICENSE)
- [nepfaff/scenesmith on GitHub](https://github.com/nepfaff/scenesmith)
- [Project website](https://scenesmith.github.io/)
- [README](https://github.com/nepfaff/scenesmith/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nepfaff-scenesmith