GibsonEnv: a simulator built from scanned real spaces

Scanned interiors instead of hand-built ones

Gibson is a virtual environment simulator that offers real-world experience for learning perception. The motivating argument in the project documentation is a specific one: learning active perception and sensorimotor control in the physical world is cumbersome because existing algorithms are too slow to learn in real time and robots are fragile and costly, which is why learning happens in simulation, which in turn raises the question of transferring to the real world.

The answer Gibson gives is that the scene content comes from the real world rather than from an artist or a game engine. The environment is named after James J. Gibson, the author of Ecological Approach to Visual Perception from 1979, and the name is a statement of method: perception and action are tied together, and the quote the project uses makes the point that you must perceive in order to move and also move in order to perceive.

The audience is research teams working on reinforcement learning, perception and robot control, and the accompanying paper, Gibson Env: Real-World Perception for Embodied Agents, appeared at CVPR 2018 as a Spotlight Oral. The repository is the distribution channel for the environment, and the technical detail sits on the project website at gibsonenv.stanford.edu.

Goggles and Bullet are the two mechanisms to look at

The project states three primary characteristics, and two of them are mechanisms rather than slogans. The first is being from the real world and reflecting its semantic complexity through virtualising real spaces. The second is having a baked-in mechanism for transferring to the real world, named the Goggles function. The third is embodiment of the agent, making it subject to the constraints of space and physics by integrating a physics engine, Bulletphysics.

Goggles is the part that distinguishes Gibson from a dataset player. A dataset gives you images and a task; a simulator gives you a closed loop where actions change what the camera sees. Goggles is the intended bridge for a policy that was learned inside the simulator and then has to survive on a real machine, and the fact that it is described as baked into the environment rather than as an add-on script tells you the transfer path is a first class part of the design.

Physics comes from Bullet, exposed to Python through the pybullet package, so the agent interacts with the scene under real constraints rather than by moving a camera through a static panorama. The examples directory in the tree is split into configs, demo, ros and train, which maps onto the four ways people use the system: configuring a scene, running the interactive demo, connecting to ROS, or training a policy.

572 spaces in the database, 39 in the core assets

The full Gibson database includes 572 spaces and 1440 floors, and the download instructions live in gibson/data/README.md. Because the core assets download package would otherwise be heavy, the release includes a small subset of 39 spaces. Users who want more download the rest and add them to the assets folder, and the gibson/assets folder is where the data needed to run the environment lives, including agent models and environments. Additional environment files can be placed in gibson/assets/dataset to run the simulator on more environments.

The core assets are distributed as a single archive, assets_core_v2.tar.gz, hosted on Google Cloud Storage, and the repository ships a download.sh script that fetches the assets file and decompresses it into the gibson/assets folder.

The project also integrated Stanford 2D3DS and Matterport 3D as separate datasets, so a user who prefers one of those scene sources can drive the same simulator with them. There is a licensing wrinkle here that code licences usually miss: the instructions say to sign the licence agreement before using the Gibson database, and the MIT licence in the LICENSE file covers the code, not the scene data.

Docker install: pull xf1280/gibson:0.3.1 and mount the dataset

There are two installation methods, and the Docker image is the recommended one. Before anything else, the documented check is:

docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

The requirements for the Docker path are Ubuntu 16.04, an Nvidia GPU with VRAM greater than 6.0GB, an Nvidia driver of 384 or newer, CUDA 9.0 or newer, and CuDNN v7 or newer. Building from source instead relaxes the versions to Ubuntu 14.04 or newer, driver 375 or newer, CUDA 8.0 or newer and CuDNN v5 or newer, but the system requirements for the GPU memory are the same.

The recommended install pulls a prebuilt image and mounts the dataset directory into the container:

docker pull xf1280/gibson:0.3.1
xhost +local:root

The full run command binds the X11 socket, passes the display through, and mounts the host dataset folder onto /root/mount/gibson/gibson/assets/dataset. The dataset is not baked into the image, which keeps the image slim, so that mount is required every time you start a container.

Building your own image is the documented alternative, and the Dockerfile does the heavy lifting:

FROM nvidia/cudagl:9.0-runtime-ubuntu16.04

The build clones the repository, runs the download script, builds with build.sh and installs the package with pip install -e .

Headless use means x11vnc on top of an X server

Gibson Env supports deployment on a headless server with remote access through x11vnc, and the documentation spells out the four steps. Install nvidia-docker2 dependencies following the starter guide and install x11vnc with sudo apt-get install x11vnc. Have an X server running on the host and run x11vnc on DISPLAY :0. Start the container with the runtime and the X socket mapped, for example mounting /tmp/.X11-unix/X0 into the container, and then run Gibson with a Python example or training script.

This tells you something about the renderer's design. The environment is a graphical application that expects an X display, not a headless offscreen renderer, so even a training job on a server needs a display to point at. That is a real operational tax for anyone running large numbers of parallel environments on a cluster, and it is a consequence of the 2018 design rather than a configuration mistake on the reader's side.

The container image also has to be built with the right graphics stack for this to work at all, which the Dockerfile lists explicitly: libglew-dev, libglm-dev, libassimp-dev, xorg-dev, libglu1-mesa-dev, libboost-dev, mesa-common-dev, freeglut3-dev, libjpeg-turbo8-dev, wmctrl and xdotool.

The pinned stack dates the project to 2018

The dependency list in setup.py is where the age shows. The install requirements pin gym==0.9.4, pybullet==1.9.4, torchvision==0.2.0, numpy, pyglet, Pillow, PyYAML, transforms3d, tqdm, pyzmq, matplotlib, mpi4py, cloudpickle, pygame, opencv-python and aenum, and the Dockerfile goes further back with a conda environment running Python 3.5, a torch 0.3.1 wheel for CUDA 9.0 and tensorflow 1.3.

Those versions are from a different era of Python tooling. Anyone arriving with a current deep learning environment will find that gym 0.9.4 does not match the modern gym API, that torchvision 0.2.0 pins an old PyTorch pairing, and that Python 3.5 in the image cannot install current packages. The practical consequence is that the Docker image, which is the recommended path and the one the project tests, is the only route that works out of the box, and building from source means recreating a 2018 dependency environment by hand.

The release history explains the shape of the project. The tags are v0.1.0, a Beta release from 2018-02-26, and v0.3.1, the 0.3.1 Release from 2018-08-11, and the documentation states that this is the 0.3.1 release with a change log in misc/CHANGELOG.md. The last push was on 2026-10-20 and the repository is not archived, so the code is still being touched, but the versioned surface has not moved past 0.3.1.

Where Gibson is the wrong tool

Three situations rule it out. If your task is computer vision on static scenes and you do not need a physics engine, a closed action loop or a transfer path, then the extra machinery buys nothing: the project itself points at Stanford 2D3DS and Matterport 3D as separate datasets, and using one of those directly is the simpler path than emulating an agent in it. If your method needs current reinforcement learning tooling, the pinned gym and torch versions make the friction higher than the payoff. And if you cannot get an Nvidia GPU with more than 6GB of VRAM, the stated minimum is not met.

There is a fourth, more practical limit. The licence story is split. The code is MIT, which is easy, but the database requires a signed licence agreement before use, and the full scene collection of 572 spaces and 1440 floors is a large download with a separate distribution story. An organisation that has already committed to a scene dataset, and to its own data handling rules, has little reason to take on a second one.

Maintenance status, licence split and what to check first

The repository is not archived and the last push was on 2026-10-20, which tells you the environment still receives attention even though the newest tagged release, 0.3.1, dates from 2018-08-11. For a project of this age that gap is the central upgrade question: there is no newer packaged environment to move to, so any change you make is a change to the 0.3.1 code base itself. The changelog at misc/CHANGELOG.md is the file to read first, because it is where the project records what changed in the release you would be building on.

The licensing is straightforward for code and less so for data. The repository is MIT licensed, so you can read, modify and redistribute the environment code under that grant, subject to the usual conditions of the licence. The scenes are separate: the database carries its own licence agreement that the download instructions ask you to sign, and those terms, not the MIT file, govern the 572 spaces and the 1440 floors. If your work is funded or published under terms that restrict how data can be used, that is a conversation with whoever owns the data, not one to resolve from the LICENSE file.

Two more practical checks before committing. The system requirements differ between the Docker and source paths, CUDA 9.0 with CuDNN v7 for the image versus CUDA 8.0 with CuDNN v5 for a source build, and the machine needs an Nvidia GPU with VRAM above 6.0GB either way. The example scripts under examples/train are the fastest way to see what a working training loop looks like, and the ROS examples under examples/ros show how the environment is wired to a robot stack for the Goggles path.

Editorial conclusion

Adopt GibsonEnv if your research needs an embodied agent trained on photorealistic scanned interiors with a built in path from simulation to a real robot, and if you can provide a machine with an Nvidia GPU holding more than 6GB of VRAM. Do not adopt it for a modern deep learning stack, since the pinned gym 0.9.4, torchvision 0.2.0 and Python 3.5 image will need serious rework, and do not adopt it for plain scene data work where downloading Stanford 2D3DS or Matterport 3D directly is simpler. Before starting, confirm your driver and CUDA versions meet the stated requirements, sign the database licence agreement, and check misc/CHANGELOG.md to see whether the 0.3.1 release already contains the fixes you need.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. StanfordVL/GibsonEnv on GitHub