Habitat-GS swaps the renderer under Habitat-Sim for Gaussian Splatting and keeps the dataset format

The extension changes the renderer and leaves the dataset contract alone

The stated goal is to be a non-intrusive extension of Habitat-Sim for embodied navigation in Gaussian Splatting scenes, and the intrusiveness boundary is drawn precisely. Habitat's standard scene dataset abstraction stays, so NavMesh and pathfinding stay, agent control stays, and Habitat-Lab integration stays. What changes is the rendering backbone, which is extended to support 3D Gaussian Splatting, plus a dynamic Gaussian avatar module for driving humanoid avatars.

In practice the project does three things: render splat scenes with Habitat's RGB and depth sensors, drive dynamic Gaussian avatars built from GaussianAvatar and AnimatableGaussians inside the simulation environment, and plug into Habitat-Lab for training and evaluation with the same scene dataset format.

The argument for the design is fidelity against cost. Against mesh-based simulators, the claim is photo-realistic rendering plus high-fidelity avatars at high efficiency, on the theory that putting splatting into an embodied simulator helps downstream embodied research.

The paper, project page, code, and dataset all became public in 2026-04, the scene collection was updated the same month with 65 high-quality scenes plus episodes and trajectories for training and evaluation, and the work was accepted to ECCV 2026 in 2026-06.

Habitat-Lab ships a NumPy pin you have to edit before installing

Habitat-Lab is described as optional but recommended, since Habitat-GS can run standalone for rendering and scene inspection while Habitat-Lab is what you want for task definition, training, and evaluation. The install is where the friction lives.

Before installing Habitat-Lab into the same environment you are told to edit its requirements file, changing `numpy==1.26.4` to `numpy>=2.0.0,<2.4`. Habitat-GS itself pins NumPy in that second range in both `pyproject.toml` and `requirements.txt`, so the upstream pin is a genuine conflict rather than a precaution, and the resolution is a manual edit inside someone else's repository.

cd habitat-lab
pip install -e habitat-lab
pip install -e habitat-baselines

Both are editable installs, which is the right choice when you are already patching the tree. The consequence is that your Habitat-GS environment is not reproducible from a lockfile: it depends on a file you edited after cloning, and a fresh clone of habitat-lab reinstates the conflict.

This is the single most common way a Habitat-GS install fails, and the failure surfaces late, as a NumPy ABI error in a dependency rather than as a clear message at install time.

Avatar tasks need a fork of habitat-lab, not the upstream one

The dynamic navigation tasks, described as avatar avoidance and avatar tracking, do not run against upstream Habitat-Lab. They require the project's own habitat-lab fork, and the instruction is to clone that repository instead and then perform the same NumPy edit and the same editable installs.

So there are two supported ways to get Habitat-Lab into a Habitat-GS environment, and they are not interchangeable. Upstream gives you point, image, and object goal navigation plus the vision-and-language tasks with StreamVLN and Uni-NaVid. The fork is what the avatar module needs, because the avoidance and tracking behaviour is implemented in the fork's task definitions rather than in the simulator extension itself.

That division is worth internalising before you plan an experiment. The simulator side is the splatting renderer and the avatar module. Anything to do with what the agent is asked to do, and what counts as success, lives in the lab repository and may need the fork.

If you are combining goal navigation with avatars, the honest answer from this documentation is that the fork is the path to start from, since it is a superset of what upstream offers and the same install steps apply.

Two environment variables decide CUDA and Bullet at install time

The build options are passed as environment variables to pip rather than as configure flags.

HABITAT_WITH_CUDA=ON HABITAT_WITH_BULLET=OFF pip install .

That is the recommended combination, CUDA on and Bullet off. Bullet is only needed if you intend to manipulate mesh objects inside a splat scene, in which case the same install is repeated with `HABITAT_WITH_BULLET=ON`.

Two further switches exist for other cases. `HABITAT_BUILD_GUI_VIEWERS=OFF` produces a headless build with no GUI, and `HABITAT_WITH_AUDIO=ON` enables the audio sensor.

Direct invocation of `setup.py` is no longer supported: the file is a legacy shim that emits a deprecation warning and then prints an error telling you to use pip, because the build moved to scikit-build-core and is configured in `pyproject.toml`. The build itself compiles the C++ sources in `src/` through CMake with a `RelWithDebInfo` build type, requires CMake 3.15 or newer, and writes into `build/{wheel_tag}`. The repository also ships `build.sh` and `install_deps.sh` for scripted setup.

The documented environment is Python 3.12 with CUDA 12.1 torch wheels

The environment is created with conda, and the versions are specific.

conda create -n habitat-gs python=3.12 cmake=3.27
conda activate habitat-gs

# IMPORTANT: Install CUDA-compatible torch first
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Torch is installed first, and the ordering is called out as important, because the CUDA 12.1 wheel index has to be selected explicitly rather than left to the default index. The repository clone is then taken with submodules:

git clone https://github.com/zju3dv/habitat-gs.git
cd habitat-gs
git submodule update --init --recursive

The packaging metadata is looser than the instructions. `pyproject.toml` declares `requires-python = ">=3.9"` and classifiers from 3.9 through 3.12, so 3.12 is the newest interpreter the metadata claims and 3.13 is outside what it declares. It also declares MIT licensing and lists Meta Platforms as author, reflecting that this is a fork-shaped extension of Habitat-Sim.

Pins are tight where they matter: `pillow==10.4.0` is exact, `numpy` sits in `>=2.0.0,<2.4`, `numba>=0.60.0`, `scipy>=1.13.0`, and `numpy-quaternion>=2024.0.0`.

The navigation agent reaches the simulator through an HTTP bridge on port 18911

The agent side is a separate process, and the `.env` template explains the wiring in unusually specific terms. `NAV_BRIDGE_HOST` and `NAV_BRIDGE_PORT` point at the HTTP server that hosts habitat-sim, on `127.0.0.1` and port `18911` by default, and the `nav_agent` subprocess connects there to execute movement and observation actions.

That separation is why some values are stated as constraints rather than defaults. `NAV_ARTIFACTS_DIR`, which holds `nav_status.json`, images, videos, and `session_stats.jsonl`, must be an absolute path, because the agent runs as a subprocess with an unknown working directory. A relative path there would write artefacts somewhere you did not expect.

Iteration is budgeted by `NAV_MAX_ITERATIONS`, set to 50, where one round is one LLM call cycle, alongside a per-round timeout that is truncated in the published template. The model side is `NAV_LLM_API_KEY`, `NAV_LLM_BASE_URL`, and `NAV_LLM_MODEL`, with the example pointing at Anthropic and `claude-sonnet-4-20250514`, and `NAV_LLM_MAX_TOKENS` defaulting to 4096 with a note that it rarely needs changing. The template notes the file is loaded by every entry point, including the agent, an MCP server, and a TUI.

Two logging variables, `HABITAT_SIM_LOG=quiet` and `MAGNUM_LOG=QUIET`, suppress verbose C++ and Magnum rendering output.

Scene assets are about 27 GB and arrive as a Hugging Face dataset

Assets are not bundled. They are distributed through a Hugging Face dataset and split into six categories, of which the first is the one everything else depends on.

| Category | Size | Required For | |---|------|-------------| | 1 | GS Scenes | ~27 GB | Everything, core scene assets | | 2 | Gaussian Avatars | ~3.1 GB | Avatar and motion tasks |

The scene collection has grown twice in 2026. In 2026-04 it was published with 65 high-quality GS scenes plus episodes and trajectories for training and evaluation, and in 2026-05 it was expanded by 64 InteriorGS scenes, bringing the total to 129.

For a first run, the practical question is whether you need the avatar categories at all. Goal navigation tasks only need the core scenes, while anything involving dynamic avatars needs the Gaussian avatar assets, and the motion-related example files in the repository, such as the fairmotion interface and the motion viewer, exist for that second case.

The repository also carries `DATASETS.md`, so the asset layout is documented separately from this page, and the example directory shows the range of entry points a working install exposes: a Gaussian viewer, motion and module viewers, a benchmark, an ablation test, and an instance segmentation example.

The NavMesh editor closes the loop from a splat scene to a navigable one

A splat scene is a rendering asset, not a walkable space, which is the practical gap between photorealism and an embodied task. The NavMesh editing tool in `web_tools/` is aimed at exactly that gap, and it is described as closing the loop from a raw 3DGS scene to a navigable simulation.

The chain it completes is 3DGS to NavMesh to simulation. You draw or edit the walkable area on the splat scene in your browser, bake a Habitat NavMesh from it, and then use that NavMesh for navigation inside Habitat-GS. Because the NavMesh is the same abstraction Habitat already uses for mesh scenes, the navigation side needs no new machinery once the walkable area is defined.

Doing this by hand otherwise means writing geometry for a scene that has no mesh, which is why the browser tool matters: the walkable region is defined against the visual scene you can actually see, rather than inferred from a floor plan you do not have.

The task catalogue that follows from it covers point, image, and object goal navigation on Habitat-Lab, vision-and-language navigation with StreamVLN and with Uni-NaVid, and dynamic navigation with Gaussian avatars, with a separate agent skills section for the tooling around them.

Editorial conclusion

Habitat-GS is the right tool if your navigation work needs photo-realistic splat scenes with Habitat's existing task definitions rather than a new simulator, and the fact that the scene dataset format is unchanged is what makes existing Habitat-Lab benchmarks portable to it. It is not the right tool for mesh scenes, mesh manipulation without opting into Bullet, or anyone unwilling to patch an upstream requirements file before installation. Before you start, edit the NumPy pin in habitat-lab to `numpy>=2.0.0,<2.4`, clone the dynamic fork if you need avatar avoidance or tracking, decide whether to pay the roughly 27 GB asset download, and confirm your driver matches the cu121 torch wheels the instructions assume.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. zju3dv/habitat-gs on GitHub