# PointWorld models the scene as 3D point flows, and asks you to build the data first

> PointWorld is the training and evaluation pipeline for a pre-trained 3D world model that predicts full-scene point flows from RGB-D captures and robot actions, published by NVIDIA and Stanford researchers under Apache-2.0. It suits a robotics researcher with a GPU cluster and a DROID or BEHAVIOR dataset to hand, and it splits data preparation from training across two branches.

**NVlabs/PointWorld** — PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

- Repository: https://github.com/NVlabs/PointWorld
- Website: https://point-world.github.io/
- Stars: 552 · Forks: 42
- Language: Python
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/nvlabs-pointworld

## The model predicts point flows, not pixels or meshes

PointWorld is described as a large pre-trained 3D world model that predicts full-scene 3D point flows from partially observable RGB-D captures and robot actions, with the robot actions also represented as 3D point flows.

That representation choice is the interesting part of the design. Instead of producing a video frame or a mesh, the model outputs a per-point displacement field over the observed scene. A robot arm movement becomes a set of point flows, the scene's response to that movement becomes another set of point flows, and the prediction target is the second.

The advantage is that the output stays in 3D and in the geometry of what the camera actually saw, so an action expressed as a point flow can be applied to the same space as the resulting scene motion. The cost is that the method needs depth, not just colour, which is why the input is described as RGB-D captures rather than RGB video.

The repository is a pipeline rather than a single model file. The top level holds train.py, eval.py, arguments.py, robot_sampler.py, scene_featurizer.py, transform_utils.py and utils.py, along with the pointworld/ package, a ptv3/ directory, training/, evaluation/, dataset_components/, visualization/, environments/, assets/, stats/ and third_party/. The presence of ptv3/ points at the Point Transformer V3 backbone being used as the architecture, and the timm pin in the setup notes is for PTv3 DropPath.

The work is credited to researchers at Stanford University and NVIDIA, with the paper on arXiv and a project website at point-world.github.io.

## Two branches, and the order you have to follow

The first section of the README is called Important Notes, and it exists because the repository is split in a way that will waste your time if you ignore it.

The `main` branch is the training and evaluation code branch for the release, and the `data` branch is the dataset preparation pipeline. The instruction is explicit: prepare the data using the `data` branch, then return to `main` for training and evaluation.

That ordering is a real constraint rather than a formality, because the released artefacts are not in the format the training code consumes. The main branch consumes local WebDataset shards and model checkpoints, and it does not consume the packaged Hugging Face archives directly.

So the pipeline is three stages. You download a packaged H5 and JSON archive from Hugging Face, you restore it with the script that comes inside the dataset repository, and then you convert it to WebDataset shards with the scripts on the data branch.

Each dataset repository includes `recover_dataset_from_parts.sh`, and the README names the two scripts you run on the data branch afterwards, data_integrity_check.py and convert_wds.py. The names tell you the shape: an integrity check before a conversion, so a truncated download is caught before it becomes a training set.

## Environment setup, and why each pin is there

The main branch provides a self-contained conda setup with no local editable dependencies, and the recommended baseline for reproducibility is Linux x86_64, Python 3.10, and an NVIDIA driver compatible with CUDA 12.4 wheels.

```bash
# from repo root
conda env create -n pointworld-env -f environments/train_eval.yml
conda activate pointworld-env
# used by the dataset/checkpoint download commands below
python -m pip install huggingface_hub==0.26.2
# timm is used for PTv3 DropPath; install without pulling extra transitive deps
python -m pip install timm==1.0.19 --no-deps
# flash-attn imports torch at build time, so install it after the base env exists
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
# keep urdfpy-compatible graph deps on a Python 3.10-safe networkx release
python -m pip install networkx==3.4.2 --no-deps
```

Every one of those pip lines carries a comment explaining the constraint, and the constraints are the interesting part. timm is pinned and installed with --no-deps so it does not drag in transitive dependencies that would move the versions the environment depends on. flash-attn is installed with --no-build-isolation because it imports torch at build time, so it cannot be installed before the base environment exists. networkx is pinned to a release that stays compatible with urdfpy on Python 3.10.

Optional visualization extras live in a second environment file, and the dependency layout section names the canonical base list at environments/requirements.txt and the extras at environments/train_eval_viz.yml, which adds matplotlib, open3d and viser.

The naming of these extras matters for a practical reason. open3d is a heavy dependency with its own conflicts, which is why it is kept out of the training environment rather than added to it.

## DINOv3 access is gated by email, and that is a scheduling risk

The third-party dependency section covers DINOv3, and it is the one step in the setup you cannot simply run.

The instruction is to request access via the official DINOv3 release page first, then use the provided download URL.

```bash
git submodule update --init --recursive
mkdir -p third_party/dinov3/checkpoints
wget -O third_party/dinov3/checkpoints/<dinov3_vitl16_pretrain_*.pth> \
  "<URL_FROM_DINOV3_ACCESS_EMAIL>"
```

The download URL arrives by email, which means the beginning of your setup depends on a human response from a third party. On a paper deadline or a cluster booking, that is the step that determines whether the rest of the pipeline can start on time.

The repository is arranged for it. There is a .gitmodules file, so DINOv3 arrives as a submodule, and third_party/ is where the checkpoint is placed under checkpoints/ with the ViT-L/16 pretrain filename. The repository also carries THIRD_PARTY_LICENSES.md, which is where the terms for the vendored components are recorded.

This is worth flagging as a dependency risk for any adopter. The core model is published under Apache-2.0 and the code is open, but a required backbone checkpoint sits behind an access request, and the pipeline does not run without it.

## The datasets are large, and the size shapes your plan

Two generated dataset packages are published on Hugging Face, the DROID generated H5 package and the BEHAVIOR generated H5 package, and the pretrained PointWorld checkpoints are hosted separately under nvidia/PointWorld_models.

The scale is stated plainly: DROID full-dataset restoration is multi-terabyte scale, but the DROID flow package is split into independent shards. You can restore a small subset of DROID flow shards, or BEHAVIOR task packages, for development and smoke testing.

That sentence is the practical instruction for anyone evaluating the project. You do not need the full DROID dataset to find out whether the code works. You restore one shard, convert it, and run a training step against it.

The destination layout is fixed by convention, and the README is clear that the arguments.py defaults follow it under LOCAL_DATASET_DIR, with `droid` pointing at `${LOCAL_DATASET_DIR}/droid/wds` and `behavior` at `${LOCAL_DATASET_DIR}/behavior/wds`.

The DROID WDS path is /path/to/droid/wds and the BEHAVIOR WDS path is /path/to/behavior/wds. So the local layout is a dataset name directory containing a wds subdirectory, and the argument defaults point there.

Two robotics datasets are involved, and they are not interchangeable. DROID is a large robot manipulation dataset, BEHAVIOR is a household activity dataset, and the two are used as separate conditions rather than merged into one corpus.

## What is missing from the release, and what that means for evaluation

The repository has no GitHub releases, so there is no published version to pin. The last push to the repository was on 2026-05-17, and the version information you have is whatever commit you are sitting on.

The README's table of contents lists sections for Training, Evaluation, Visualization, Known Limitations and Contributing, and the excerpt available here stops partway through the datasets and checkpoints chapter. So the specifics of how to launch a training run and what the evaluation harness measures are described in the README rather than in the excerpt available here.

What the repository layout does tell you is the separation of concerns. training/ and train.py are the training side, evaluation/ and eval.py the evaluation side, visualization/ the rendering, dataset_components/ the data pipeline pieces, and stats/ presumably the dataset statistics that a normalising pipeline needs. robot_sampler.py is a distinct module, which suggests the robot motion sampling is separable from the scene model, and scene_featurizer.py is separate again for turning raw observations into model inputs.

The honest reading is that this is a research pipeline with a fixed expected shape of data rather than a library with a stable API. arguments.py at the root holds the configuration, and the directory convention under LOCAL_DATASET_DIR is part of that contract: if your data is not laid out that way, the defaults will not find it.

## Where PointWorld is the wrong tool

Three cases argue against it.

The first is evaluation without a cluster. The setup asks for an NVIDIA driver compatible with CUDA 12.4 wheels, and the model is a large pre-trained 3D world model over point flows. Add the multi-terabyte dataset restoration and the flash-attn build step, and this is not a project you try in an afternoon.

The second is anything outside manipulation research. The datasets are DROID and BEHAVIOR, the input is RGB-D, and the actions are point flows. If your problem is autonomous driving, or grasping with a different action space, or manipulation from monocular video, the representation choice that makes PointWorld elegant also puts it out of reach.

The third is a need for a stable, versioned dependency. There are no GitHub releases, and the code sits on a research branch layout where the data preparation and the training live on separate branches. A paper reproduction is a good fit for that arrangement. A production service that needs a pinned commit, a supported upgrade path and a predictable environment is not.

The alternative in the same direction is training a smaller world model on a single dataset without the released checkpoints, using an existing 3D representation library. The difference in approach is that PointWorld commits to a point-flow formulation over the whole scene and publishes the data and weights to go with it, which means you can reproduce the reported behaviour rather than approximate it, at the cost of the multi-terabyte pipeline and the gated DINOv3 checkpoint.

## Conclusion

PointWorld fits a robotics research group with CUDA hardware, multi-terabyte storage and an appetite for the data pipeline, since the model, the datasets and the checkpoints are all published and the setup is pinned to specific versions. It does not fit a quick evaluation, because the datasets must be downloaded, restored, checked and converted to WebDataset shards before a training run can start. Prepare the data on the `data` branch first, then switch to `main`, and confirm the DINOv3 checkpoint access has been granted before you reserve a cluster, since that download URL only arrives by email.

## FAQ

### What is PointWorld?

PointWorld is a large pre-trained 3D world model that predicts full-scene 3D point flows from partially observable RGB-D captures and robot actions, with the actions also represented as 3D point flows. The repository is the training and evaluation pipeline for it.

### How do I set up the PointWorld environment?

Create the conda environment from environments/train_eval.yml and activate it, then pip install huggingface_hub==0.26.2, timm==1.0.19 with --no-deps, flash-attn==2.7.4.post1 with --no-build-isolation, and networkx==3.4.2 with --no-deps. The recommended baseline is Linux x86_64 with Python 3.10 and a driver compatible with CUDA 12.4 wheels.

### Where do the PointWorld datasets and checkpoints come from?

The generated H5 packages for DROID and BEHAVIOR and the pretrained checkpoints are hosted on Hugging Face under the nvidia organisation. Each dataset repository includes recover_dataset_from_parts.sh, which restores the download before you run data_integrity_check.py and convert_wds.py on the data branch.

### Why does PointWorld need access to DINOv3?

DINOv3 is a third-party dependency, requested through the official DINOv3 release page, and the checkpoint is fetched with wget into third_party/dinov3/checkpoints after the submodule is initialised. Because the download URL arrives by email, that request is the step to start first.

### What licence is PointWorld released under?

Apache-2.0, with the SPDX identifier recorded in the README header and the LICENSE file at the repository root. Third-party component terms are recorded separately in THIRD_PARTY_LICENSES.md.

## Sources

- [Issues](https://github.com/NVlabs/PointWorld/issues)
- [License: Apache-2.0](https://github.com/NVlabs/PointWorld/blob/main/LICENSE)
- [NVlabs/PointWorld on GitHub](https://github.com/NVlabs/PointWorld)
- [Project website](https://point-world.github.io/)
- [README](https://github.com/NVlabs/PointWorld/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/nvlabs-pointworld