VirtualHome: a Python API that drives a Unity household simulator and a pure-Python graph engine
API to run VirtualHome, a Multi-Agent Household Simulator
At a glance
- What is it?
- VirtualHome turns household activities into programs and environments into graphs, then renders them either through a downloaded Unity executable or through an in-process graph simulator. The Python package is the easy half; the Unity binary is a separate download with a display or X server requirement.
- Who is it for?
- Adopt VirtualHome if you need scripted, program-level household activity data (action sequences plus environment graphs) and can tolerate the Unity executable as an external download that needs a display or an X server such as the helper_scripts/startx.py path. Do not adopt it if you need photorealism, because the README lists that under Currently in Development, or if you cannot run a graphical executable at all.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 119 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap VirtualHome fills: scripted household activity instead of raw pixels
Most embodied AI environments hand you an observation and an action space and let you learn from scratch. VirtualHome starts one level up. An activity is written as a program, a sequence of instructions, and the environment is written as a graph. The simulator executes the program and returns either a video or a sequence of graphs showing how the environment changed. That means the primary artifact is structured and inspectable: you know which action was attempted, in which room, on which object, and what the resulting state was. The README describes the platform as simulating complex household activities via programs, with interactions such as picking up objects, switching appliances on and off, and opening appliances. The intended audience is researchers working on embodied AI and multi-agent activity simulation, plus anyone who needs ground-truth streams (timestamped actions, instance and semantic segmentation, optical flow, depth) without hand-labelling them.
Two simulators, one program format, and where they diverge
The Overview section splits the system in two. The Unity Simulator is built in Unity and generates videos of activities; it requires downloading a platform executable and driving it through the Python API in virtualhome/simulation/unity_simulator/. The Evolving Graph simulator runs fully in Python and produces a sequence of graphs as the program executes; it lives under virtualhome/simulation/evolving_graph. The two are not interchangeable. The README states plainly that some objects and actions in the Evolving Graph are not supported yet in the Unity Simulator. That sentence is the single most important line in the document for anyone planning a project, because it means the cheaper, headless, no-download path and the visualisation path do not cover the same action set. If your activity depends on an object that only exists in the graph simulator, you will get graphs but no video. The data flow is consistent across both: program plus graph in, execution trace out, rendered as video or as evolving graphs. Multi-agent activity is supported, and the README also notes an OpenAI Gym-like environment in unity_environment.py for RL training, plus an example combining environments with Ray.
Getting it running: pip, the executable, and the display problem
The Python side is one command: pip install virtualhome. The simulator side is a manual download. The README points to three archives for v2.3.0 (linux_exec.zip, macos_exec.zip, windows_exec.zip) and instructs you to move the executable under simulation/unity_simulator. To test locally you either double-click it or run it from a terminal, and the README recommends windowed mode: ./path_to_exec -screen-fullscreen 0 -screen-quality 4. Then you run demo/unity_demo.ipynb. The interesting case is headless use. The README offers two routes: Docker, or an X server, linking to an external Medium post for the X server instructions. For Linux with an X server, you start it with sudo python helper_scripts/startx.py $display_num, then launch the executable on a separate terminal with DISPLAY=:display_num ./{path_sim}/{exec_file}.x86_64 -batchmode. Alternatively, UnityCommunication can launch the executable itself: comm = comm_unity.UnityCommunication(file_name=file_name, port={your_port}, x_display={your_display}). Note that startx.py is invoked with sudo and takes a display number, so this is not a sandboxed, unprivileged setup. The README also mentions opening multiple executables, which implies port management is your responsibility when you scale out.
What the release notes tell you about maturity and cost
The most recent tagged release is v2.3.0 from March 2022, following v2.2.0 in March 2021 and v2.1.0 in January 2021. The 2.3 notes list procedural generation of environments, more custom environments, enhanced simulated physics, a time management system with synchronized day and night, new outdoor terrain with sunlight and shadows, improved indoor realtime lighting, more realistic rooms, and performance and stability work. The repository itself has been pushed more recently than the last tag, so the tagged releases are not a complete picture of the code state. The upgrade cost is asymmetric. Upgrading the Python package is a pip install. Upgrading the simulator means re-downloading the matching executable for your platform and re-testing your scripts against it, because the API talks to a binary whose version is pinned by URL. The README's download links are version-specific (v2.3.0), which means an old pinned URL is a hard dependency in any reproducible build. Procedural generation, listed as both new in 2.3 and still under development, is the feature most likely to change shape between versions.
Where it breaks: the executable is the fragile part
The Python package installs cleanly, but nothing in the README suggests the Unity executable is optional if you want video. If you have no monitor and cannot run Docker or an X server, the visual side is unavailable to you. The -batchmode flag is documented for the X server path, not as a general headless solution, so treat it as tied to that setup. The helper script is run with sudo, which is a poor fit for shared clusters where you do not control the display stack. There is also the action-coverage mismatch already noted: the Evolving Graph supports objects and actions that Unity does not, so a pipeline that develops against graphs and then renders in Unity can fail at the render step. Finally, the README's Currently in Development list includes photorealism, more actions and object interactions, and human interaction. Those are explicitly unfinished, so any evaluation that depends on realistic rendering or on human characters is building on stated work-in-progress. The README does not document frame rates, memory ceilings, or how many agents a single executable instance can host, so capacity planning has to be empirical.
The alternative: Habitat or AI2-THOR for rendering-first work
If your goal is visual navigation or photorealistic perception, the natural comparison is Habitat or AI2-THOR. The difference is where the abstraction sits. VirtualHome's unit of authoring is a program plus a graph, and the README's framing is that the simulator executes the program and returns a video or a sequence of graphs. Habitat and AI2-THOR are built around an agent acting in a 3D scene with sensor observations as the primary interface. VirtualHome can produce segmentation, depth and optical flow, but the README positions those as streamed ground truth alongside a scripted activity, not as the main loop. Conversely, if you want to specify a 30-step tea-making procedure and get a labelled trace of every state change, writing that as a VirtualHome program is more direct than scripting an agent to discover the same sequence. The trade-off is real in both directions: VirtualHome gives you activity structure and multi-agent scripting at the cost of a heavier, less flexible rendering stack; a rendering-first simulator gives you fidelity at the cost of having to express activities as policies or trajectories rather than programs.
Licence and the practical shape of maintenance
The repository is MIT-licensed, which is permissive and places few conditions on reuse beyond preserving the notice. That covers the code in the repository. It does not automatically tell you the terms attached to the downloaded Unity executables, which are distributed from virtual-home.org rather than through the package, and the README does not state their licence. If you redistribute a container that bundles the executable, that is the thing to check, and I am not giving legal advice here, only pointing at the gap. On maintenance: the Python package and the binary version together form the compatibility surface. A pip upgrade that outpaces your downloaded executable, or a pinned v2.3.0 URL that stops resolving, are the two failure modes to plan for. Keeping the executable archive in your own artifact store, rather than fetching from virtual-home.org at build time, removes the second one.
Who should pick this up, and what to check on day one
VirtualHome fits teams that need structured household activity data, multi-agent scripting, or an RL environment with a Gym-like interface, and that already have a way to run a graphical executable (a workstation, a Docker setup, or an X server). It does not fit teams that need photorealism today, since the README lists it as in development, or teams that cannot run a display at all and cannot use Docker. The first thing to verify is coverage: take the specific actions and objects your activity needs and confirm they behave in the Unity Simulator, not just in the Evolving Graph, because the README says some are not yet supported in Unity. The second is the launch path on your target machine: run the executable with -screen-fullscreen 0 -screen-quality 4 locally, and if you are on a headless box, confirm helper_scripts/startx.py works with your display number before you build anything on top of comm_unity.UnityCommunication. If both checks pass, the pip install is the least of your problems.
Editorial conclusion
Adopt VirtualHome if you need scripted, program-level household activity data (action sequences plus environment graphs) and can tolerate the Unity executable as an external download that needs a display or an X server such as the helper_scripts/startx.py path. Do not adopt it if you need photorealism, because the README lists that under Currently in Development, or if you cannot run a graphical executable at all. Before committing, verify which actions your target environment supports in the Unity Simulator versus the Evolving Graph, since the README states some objects and actions are not yet supported in Unity, and check that the v2.3.0 executable for your platform launches with -screen-fullscreen 0 -screen-quality 4.
Community notes