HA-VLN 2.0: benchmarking agents around moving people

A benchmark that treats people as obstacles with intentions

Classic vision-and-language navigation asks an agent to follow a sentence such as go past the kitchen counter and stop by the sofa. The scene is static and the sentence is about furniture. HA-VLN 2.0 changes both halves of that setup. Instructions describe human activity, for example an individual engaged in a phone conversation while pacing back and forth, or a group of friends raising their glasses in a toast, and the agent has to reach a goal without walking through those scenes. That turns navigation into a social problem: the shortest path is often the socially worst one. The project is a research benchmark rather than a product, and it targets researchers who train or evaluate multimodal agents, plus the challenge organisers now running an HA-VLN track in the RoboWorld Challenge 2026. The repository is written primarily in C++ for the simulator side, with Python packages for the agent side, and it is published under an MIT licence, which is permissive for both academic and commercial work as long as the notice travels with the code.

Two task settings share one benchmark

The headline contribution is that the benchmark covers discrete and continuous environments at once. The README names them HA-VLN-DE and HA-VLN-CE, and the technical report on arXiv 2503.14229 describes the 2.0 release as unifying discrete and continuous human-aware navigation. A discrete agent acts on a graph of viewpoints, choosing one reachable node at a time, while a continuous agent has to output motion at a finer granularity. Both versions of the task are scored against the same kind of goal instruction, and both are evaluated on social awareness, which is the metric that separates this benchmark from plain R2R-style evaluation. The release also introduced HAPS 2.0 alongside the benchmark, and the news entry for 2025-03 says the unification covers social-awareness evaluation across 16,844 instructions. That scale is what turns it into a leaderboard problem rather than a demo: a model has to generalise across instruction phrasings rather than memorise a handful of routes.

HASimulator renders the humans you are meant to avoid

The simulator sits at HASimulator/ in the repository and is described as enabling real-time rendering of human activities together with unified APIs for navigation development. Two capabilities follow from that. First, because the environment renders rather than replays, the same episode can be evaluated repeatedly with different model behaviour, which is what a leaderboard needs. Second, the human figures come from the HAPS 2.0 dataset with detailed 3D human motion models, so the agent sees articulated people at varying poses rather than a static sprite. The repository also ships a human-scene fusion stage, listed separately from the simulator in the table of contents, which is how those rendered bodies are placed into the room the agent is looking at. The top-level layout separates the pieces cleanly: HASimulator/ for the environment, Data/ for the datasets, agent/ for the reference models, scripts/ for tooling, and demo/ for recorded clips.

Two datasets, one of them built around complex instructions

The data ships in two directories under Data/. HAPS 2.0 is the Human Activity and Pose Simulation dataset, carrying the 3D human motion models that drive the rendered behaviour. HA-R2R is the HA Room-to-Room dataset, and its selling point is instruction complexity: the instructions weave together the route and the social situation at each step, so a single sentence can mention a person ironing clothes in a hallway, a reader in a bedroom, and the final position near a sink. The README shows expanded examples of exactly this, with each social clause highlighted in the navigation instruction. The dataset is published in two places, a HuggingFace dataset page at fly1113/HA-VLN and a Google Drive folder, which is a convenience and a warning: expect a large download, and expect to place the files yourself into the directory layout the scripts expect. The README does not document the on-disk structure in the copy available here, only that a Dataset Organization section exists.

HA-VLN-VL and HA-VLN-CMA differ in where attention goes

The agent directory holds two reference models, and the contrast between them is the clearest way to understand the research question. HA-VLN-VL is the vision-and-language model, proposed to address the visual-language understanding problem: reading an instruction whose clauses describe social scenes and connecting them to what the camera sees. HA-VLN-CMA is the Cross-Model Attention model, proposed to address dynamic decision-making: deciding what to do next when the people in the scene are moving and the plan has to adapt. Both live under agent/, so a comparison run does not require reaching into two separate repositories. If you are evaluating, run both and treat the gap between them as the measure of how much dynamic social reasoning a method adds over plain instruction grounding. The README also points to the project webpage and leaderboard, which is where the official numbers are published.

Getting the environment and the data in place

The Quick Start section describes a flow of downloading the datasets, setting up a Docker environment and reproducing the published results, but the instructions in the repository copy available here stop at that sentence, so the exact shell steps are not spelled out. The dependency lists are published, though, in requirements.txt at the root and in the Python 3.8 variants requirements-py38.txt and requirements-dino-py38.txt. The pinned core of the root list looks like this:

setuptools==59.5.0
torch>=1.6.0
gym>=0.17.3
numpy>=1.16.1
yacs>=0.1.5
Opencv-python>=3.3.0
tqdm>=4.0.0
lmdb
jsonlines

The rest of the file adds data-handling utilities, including dtw==1.4.0 and fastdtw==0.3.4 for trajectory comparison, along with gdown, tb-nightly for tensorboard logging, networkx, msgpack_numpy, simplejson, pre-commit and braceexpand. Note the pin on setuptools and the presence of two Python 3.8 specific requirement files: this is a research stack built around older tooling habits, not a freshly packaged project. Datasets come from the HuggingFace dataset page fly1113/HA-VLN or from the Google Drive folder, and both the page and the drive folder are the paths the project points to for the first download.

An active repository with no release tags

There are no GitHub releases for this repository, so there is no versioned artifact to pin and no upgrade path through the releases page. Instead you follow the main branch, which means the dependency pins and the simulator code can move under you between runs. The last push was on 2026-10-13, and the project has been busy since the 2.0 release in 2025-03: HA-VLN 2.0 was accepted to IROS 2026 in 2026-06, HA-VLN 1.0 was accepted to the NeurIPS 2024 Datasets and Benchmarks Track in 2024-09, and an HA-VLN track runs inside the RoboWorld Challenge 2026 with the RoboPAD Workshop at NeurIPS 2026. The 2025-08 news entry mentions a substantial upgrade of the repository and a small internal competition, which suggests the code was reorganised after the initial release. For a leaderboard submission, that combination of an active main branch and no tagged releases means you should record the exact commit you evaluated.

Editorial conclusion

HA-VLN 2.0 is worth your attention if you build vision-and-language agents and want to measure whether they respect people in a scene, since it ships a simulator, both task variants, two datasets and two reference models under one MIT licence. It is the wrong tool for a robotics team that needs photoreal human motion captured from real footage, and it is also a poor fit if you cannot host the Docker environment and download the released datasets. Before committing, check that your stack matches the pinned Python dependencies, decide whether the discrete or continuous split matches your deployment, and read the Quick Start section on the project page, because the repository's own setup instructions are cut short in the copy available here.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. UWMILab/HA-VLN on GitHub