# SimWorld: an Unreal Engine 5 city behind a gym-like API

> A simulator for LLM and VLM agents built on Unreal Engine 5, where the Python package is a client rather than the world itself. It ships two city scenes by default, needs a real GPU, and its packaging metadata contradicts both its license file and its own system requirements.

**SimWorld-AI/SimWorld** — SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds

- Repository: https://github.com/SimWorld-AI/SimWorld
- Website: https://simworld.org/
- Stars: 791 · Forks: 90
- Language: Python
- License: Apache-2.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/simworld-ai-simworld

## The base package is two city scenes and an empty map

The usage model is tiered, and the default tier is deliberately small. The Base package contains two lightweight city scenes plus one empty map, and that is enough for core agent interaction and quick testing. If a scenario needs more, an optional expansion adds more than a hundred pre-built maps.

The third tier is where you bring your own material: your own Unreal Engine environments, assets and agent models can be imported into SimWorld for fully customized simulations. The mechanism is Unreal's `.pak` packaging, so a custom world arrives as a file the simulator can load rather than as a patch to the Python code.

That three-level structure is what makes the project usable at different budgets. A new user clones and runs against two scenes, which is small enough to iterate on. A benchmark author pulls the larger map set. A lab with its own art pipeline ships a pak file. The cost is that you are always working against a fixed simulator rather than something you can reshape in Python.

The stated purpose is developing and evaluating LLM and VLM agents in physical and social environments, with realistic physics, language-based procedural generation, multi-modal perception, natural language actions, and scenarios for navigation, planning and strategic cooperation.

## The hardware requirement, not the API, is the adoption barrier

The system requirements are the first thing to read. SimWorld needs Windows or Linux, a dedicated GPU with at least 6GB of VRAM, 32GB of RAM, and somewhere between 50GB and 200GB of disk depending on which package you install. Detailed device recommendations live in a separate installation page.

The range in that disk figure is the honest signal. Fifty gigabytes is the small end and two hundred is the large end, and the difference is the map and asset packages rather than the Python client. So the sizing question is really a question about how many of the optional environments you intend to load, and installing everything on a machine that only needs Base costs a large download for no benefit.

A dedicated GPU with 6GB is a modest requirement next to those disk numbers, but it is still a discrete card rather than integrated graphics. That single line decides who can realistically evaluate the project, since a laptop without one can install the client and discover that nothing renders.

On the Python side the requirement is 3.10 or later, and the documented client install is four commands:

```bash
git clone https://github.com/SimWorld-AI/SimWorld.git
cd SimWorld
conda create -n simworld python=3.10
conda activate simworld
pip install -e .
```

Step two of the setup is a separate download of the Unreal Engine server, which is why the client alone leaves you with nothing to run.

## The Python package is a client, not the simulator

The architecture is three layers, and knowing which one you are writing against saves a lot of confusion.

The Unreal Engine backend provides the environments, the assets and the physics simulation. The environment layer sits on top of it and adds procedural city generation, language-driven scene editing, gym-like APIs for LLM and VLM agents, and traffic simulation. The agent layer is where the model reasons over multi-modal observations and history and then executes actions through a local action planner.

That last part is the interesting design choice. The language model does not emit engine commands. It decides, and a separate local planner turns decisions into movement, which is why the package ships a `local_planner/` module alongside the `agent/` module.

The bridge between Python and the engine is its own package. `communicator/` is described as the core component for connecting Unreal Engine, and it includes UnrealCV support, so the client can also drive the engine through a computer-vision channel rather than only through a bespoke protocol. Underneath sit `map/` for maps and waypoints, `traffic/` for the traffic system, `citygen/` for procedural city layouts, and `config/` for the loader and its default config file.

Everything the Python side does is therefore a remote control for a process you also have to install and run.

## Ten modules, and two of the dependencies are a desktop GUI

The package layout is short enough to read in one pass: `local_planner/`, `agent/`, `assets_rp/`, `citygen/`, `communicator/`, `config/`, `llm/`, `map/`, `traffic/` and `utils/`.

The dependency list is more revealing than the module list. Alongside NumPy, pandas, OpenCV and Pillow, the install requires PyQt5 and pyqtgraph, which are desktop interface toolkits, and `sentence-transformers` with `faiss-cpu`, which is a sentence embedding model plus a vector index. The second pair exists for `assets_rp/`, the live editor component described as handling retrieval and re-placement of assets, which is a semantic search over an asset library rather than a filename lookup.

That also means a CPU-only vector index is what you get, `faiss-cpu` rather than a GPU build, and that installing the package pulls a transformer model stack whether or not you use that component.

The last dependency is `openai`, which is what the minimal example uses through a `BaseLLM` class instantiated with a model name, in the shipped demo with `gpt-4o`. The LLM interface is therefore a thin wrapper rather than a multi-provider abstraction, and using a different model means writing or swapping that class.

Packaging config ships as package data: `simworld.config` carries its YAML files and `simworld.data` carries JSON plus asset images.

## The minimal example is a language model that picks navigation actions

The quick tour is a navigation task with a humanoid agent, and the point of it is to show the interface rather than the capabilities. The documentation says plainly that the code is simplified for demonstration and that the complete version is in the examples notebook for the gym interface.

The imports tell you what a task touches. Configuration comes from `simworld.config`, the engine connection from `simworld.communicator.communicator` with UnrealCV alongside it, the language model from `simworld.llm.base_llm`, maps and waypoints from `simworld.map.map`, the agent body from `simworld.agent.humanoid`, and geometry from `simworld.utils.vector`.

The agent itself is a small class with two fields: a language model instance and a system prompt describing the task. Its action method takes an observation and returns a decision, and the surrounding loop feeds observations back in. There is no prompt template library, no tool registry and no memory module in the shipped example, which makes it a fair representation of the floor and not the ceiling.

The examples directory carries eleven notebooks covering the pieces individually: asset retrieval and replacement, camera control, the gym interface, layout generation with and without visualization, the local action planner, maps, traffic simulation, direct Unreal Engine commands, and world generation. That list is a better map of the project's surface than the feature summary.

## The packaging metadata contradicts the license and the requirements

Two inconsistencies in the packaging configuration are worth catching before you build on the project.

The first is licensing. The repository's license is reported as Apache-2.0, while the packaging classifiers still declare `License :: OSI Approved :: MIT License`. Nothing about the code changes, but a classifier is what tools read, and an automated dependency scanner that trusts the classifier will tell you the wrong terms.

The second is platform support. The classifiers say `Operating System :: OS Independent`, while the system requirements name Windows or Linux, a dedicated GPU with at least 6GB of VRAM, and 32GB of RAM. A pip install will succeed on a machine that cannot run the simulator, because installation does not check for a GPU.

The third item is smaller. The package description in `setup.py` calls it a simulation framework for urban environments and traffic, which undersells it: the environment layer also does language-driven scene editing and gym-style agent APIs. The version is 0.1.0 and the author field is a team name with one contact address, `python_requires` is 3.10 or later, and the development extra adds pytest, flake8 and black, with a `.flake8` config and a pre-commit configuration at the root.

None of this makes the project unusable. It does mean the metadata was written once and not revisited, which is a reason to trust the documentation site over the package index entry.

## One prototype release, and the badges still point at an old organisation

The release history is a single entry: v0.1.0, titled as a prototype implementation, published 2026-05-30. The last push to the default branch is 2026-06-26, so nothing has been tagged in the months since, and the project sits at version 0.1.0 in the packaging too.

The publication record is stronger than the release record. The project reports a demo accepted at the CVPR 2025 demonstration track in March 2025, a first formal release in June 2025, acceptance to the NeurIPS 2025 main track as a spotlight in September 2025, a white paper on arXiv in November 2025, and support for importing customized environments and agents in January 2026. A site at simworld.org hosts the white paper, and the documentation is built on readthedocs.

One detail suggests the repository was moved. The badges at the top of the README still point at `github.com/maitrix-org/SimWorld`, while the clone instructions and the current path use `SimWorld-AI/SimWorld`. Anyone following a badge lands on the old location.

Scale-wise it has 789 stars, 91 forks and 12 open issues. The fork count is high relative to the issue count, which fits a project people pull and modify rather than one they file bugs against, and that is the right expectation to bring to a simulator with an asset pipeline.

## Conclusion

SimWorld fits a research group that wants multimodal, physically plausible environments with a familiar agent interface rather than a toy text world, and the published record, a NeurIPS spotlight and a CVPR demonstration, suggests the team ships what it claims. Skip it if you have no discrete GPU or no patience for a 50 to 200 GB download. Before you plan around it, check whether the packaging classifiers still describe your setup, because they claim an MIT license and OS independence that the license file and the hardware requirements both contradict.

## FAQ

### What does SimWorld need to run?

Windows or Linux, a dedicated GPU with at least 6GB of VRAM, 32GB of RAM, and 50GB to 200GB of disk depending on the package. On the Python side it needs 3.10 or later, and the documented setup uses a conda environment.

### How do I install the SimWorld Python client?

Clone the repository, create a conda environment with python=3.10, activate it, and run pip install -e . The Unreal Engine server is a separate download in step two, so the client alone is not enough to run anything.

### What is included in the base SimWorld package?

Two lightweight city scenes and one empty map, which is enough for core agent interaction and quick testing. More than a hundred pre-built maps are an optional expansion, and your own environments arrive as .pak files.

### How does SimWorld connect Python to Unreal Engine?

Through its communicator package, which also supports UnrealCV. The agent layer reasons over multi-modal observations and executes actions through a separate local action planner rather than issuing engine commands directly.

### Is SimWorld a finished release or a prototype?

The only tagged release is v0.1.0, published 2026-05-30 and titled a prototype implementation. The project does report a NeurIPS 2025 spotlight and a CVPR 2025 demonstration track paper.

## Sources

- [License: Apache-2.0](https://github.com/SimWorld-AI/SimWorld/blob/main/LICENSE)
- [Project website](https://simworld.org/)
- [README](https://github.com/SimWorld-AI/SimWorld/blob/main/README.md)
- [Releases](https://github.com/SimWorld-AI/SimWorld/releases)
- [SimWorld-AI/SimWorld on GitHub](https://github.com/SimWorld-AI/SimWorld)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/simworld-ai-simworld