# WorldScore: a benchmark that scores world generation by camera control, not video quality

> WorldScore is the official implementation of a unified evaluation benchmark for world generation models, released under MIT by a Stanford group. It judges generated scenes against a specified camera path, which is where it departs from single-scene video quality metrics.

**haoyi-duan/WorldScore** — Official implementation for WorldScore: A Unified Evaluation Benchmark for World Generation

- Repository: https://github.com/haoyi-duan/WorldScore
- Website: https://haoyi-duan.github.io/WorldScore/
- Stars: 319 · Forks: 20
- Language: Python
- License: MIT
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/haoyi-duan-worldscore

## What WorldScore actually measures, and why camera paths are the unit of judgement

Most video generation benchmarks score a clip on its own terms: sharpness, motion smoothness, subject consistency. WorldScore starts from a different premise. The README describes an example where two models are given the same initial bedroom scene and the same camera path, "pan left" then "move left" then "pull out", and existing benchmarks rate them similarly on single-scene video quality. WorldScore separates them by finding that one model fails to generate a new scene or follow the instructed camera movement. That is the whole point of the benchmark: it treats the world as something a camera moves through, and it scores the model on whether the world holds up under that movement. The intended audience is researchers and labs that publish world generation models and want a number that reflects controllability, not just pixels. The repository is the official implementation for the paper, and the project ships a dataset on Hugging Face plus a leaderboard Space, so the benchmark is meant to be run by outsiders, not only by the authors.

## The three model categories and the config that ties them to the harness

WorldScore does not assume one kind of generator. The registration file worldscore/benchmark/utils/modeltype.py groups models into three types: threedgen for 3D scene generation, fourdgen for 4D scene generation, and videogen for video generation. Each model you want to evaluate gets a YAML config in config/model_configs, and the README shows the fields: model, runs_root, resolution, generate_type (i2v or t2v), frames and fps. The runs_root key is interpolated with OmegaConf's oc.env syntax against MODEL_PATH, so the harness reads your environment rather than a hardcoded path. Outputs land under MODEL_PATH/<model_name_repo>/worldscore_output according to the setup section. This is a deliberate design: the benchmark owns the scoring, the model repo owns the weights and the generation loop, and the two meet at a config file and a model class in world_generators. The README states there are two ways to adapt a model, and points to world_generators/README.md for the WonderJourney and WonderWorld adaptations. If your architecture does not map onto one of the three types, the harness has no slot for it, and that is a real boundary rather than a missing flag.

## Installing WorldScore and running a first evaluation

The README's setup section is short and assumes you have conda and a model repo already checked out. Start by cloning and creating the environment, then install the package in editable mode. The install pulls the dependencies listed in setup.py, which include hydra-core, omegaconf, submitit, opencv-python and python-dotenv.

```bash
git clone https://github.com/haoyi-duan/WorldScore.git
cd WorldScore
conda create -n world_gen python=3.10
conda activate world_gen
pip install -e .
```

Before anything runs, the repository expects a .env file at its root with three variables. WORLDSCORE_PATH is where the repo was cloned, MODEL_PATH is the root of the model repo and also where evaluation outputs are written, and DATA_PATH is where the dataset will live.

```sh
WORLDSCORE_PATH=/path/to/worldscore
MODEL_PATH=/path/to/model
DATA_PATH=/path/to/dataset
```

The README warns that the export step must be repeated in every new terminal session, which is easy to miss and produces confusing path errors when skipped.

```sh
export $(grep -v '^#' .env | xargs)
```

With the variables exported, download.py fetches and organizes the dataset into $DATA_PATH/WorldScore-Dataset.

```bash
python download.py
```

After that you register your model with a YAML file in config/model_configs, add its name to the matching list in worldscore/benchmark/utils/modeltype.py, and implement the model class. The package installs four console entry points: worldscore, worldscore-eval, worldscore-analysis and worldscore-gen, so evaluation and analysis can be invoked as commands rather than through ad hoc scripts. The README does not document the flags for those entry points, so read worldscore/cli.py before assuming an interface.

## Where the harness will fight you: paths, sessions and unregistered models

The environment design is the first place people lose time. Three absolute paths have to be correct at once, and MODEL_PATH doubles as both the model repo root and the output root, so a wrong value writes evaluation artifacts into the wrong tree rather than failing loudly. The export command is not persisted anywhere; the README's own note says it must be repeated in every new terminal session. There is no documented fallback that reads .env automatically. The second failure mode is registration. A model that is not listed under threedgen, fourdgen or videogen in modeltype.py will not be picked up, and the README does not describe an error message for that case. The third is scope. WorldScore is built to compare generators on instructed camera movement and scene continuation; if your goal is to score a single clip for aesthetic quality, or to evaluate a model that does not generate temporally extended worlds, this benchmark is the wrong instrument and its setup cost is not recoverable. The README also does not document rollback, version pinning for model repos, or how to reproduce a leaderboard entry locally.

## WorldScore against VBench and the newer world-model benchmarks

VBench is the reference point most people already know, and the difference is structural. VBench-style evaluation scores a generated video against a set of quality and consistency dimensions. WorldScore adds the instruction layer: the same scene is rendered under a specified camera path, and the model is judged on whether it follows that path and generates new content as the camera moves. That is why the README's example can separate two models that a quality-only metric would rank together. The trade-off is that WorldScore needs a dataset of initial scenes and camera paths, a model wrapper per architecture, and a working generation pipeline before any number appears, while a quality metric can be pointed at a folder of clips. Other benchmarks in this space, including WorldArena, 4DWorldBench and Worldmodelbench, are alternatives worth reading if your model is 4D or agent-oriented, but the repository does not compare itself to them, so treat any equivalence claim as untested.

## Licence, maintenance and what upgrading costs

The repository is MIT licensed, which permits commercial use and modification provided the copyright notice and permission notice are retained. That is permissive, but it covers the benchmark code, not the models you plug in or the dataset hosted on Hugging Face; those carry their own terms, and the README does not restate them. On maintenance, the last push was on 2026-07-23, so the project has moved recently, and the updates list shows evaluation code for WonderJourney and WonderWorld added in 11/2025 and Voyager results in 06/2025. There are no retrieved releases, so versioning runs through setup.py, which declares version 1.0.0. Upgrading means pulling main and re-running pip install -e ., because the package is installed in editable mode and the entry points come from the working tree. If you have modified modeltype.py or added configs, a pull can conflict with your registrations; keep those changes in a branch. Nothing in the README describes a migration path between benchmark versions, so pin the commit you evaluated against if you need to reproduce a score later.

## Conclusion

Adopt WorldScore if you are publishing a world generation model and need a camera-path-aware score that a leaderboard can compare across 3D, 4D and video generators. Do not adopt it if you only want a quick perceptual quality number for a single clip; VBench-style metrics cover that with far less setup. Before committing, verify that your model fits one of the three registered types in worldscore/benchmark/utils/modeltype.py, that DATA_PATH has room for WorldScore-Dataset, and that you can re-export the .env variables in every new terminal session, because the workflow breaks silently otherwise.

## FAQ

### What is WorldScore?

It is the official implementation of WorldScore, a unified evaluation benchmark for world generation, published by Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei and Jiajun Wu and accepted to ICCV 2025. It scores generated worlds against instructed camera movement rather than single-scene video quality alone.

### How do I install WorldScore?

Clone the repository, create a Python 3.10 environment, and run pip install -e . from the repo root. Then create a .env file with WORLDSCORE_PATH, MODEL_PATH and DATA_PATH and export those variables in every new terminal session.

### Where is the WorldScore dataset and leaderboard?

The README links a dataset at huggingface.co/datasets/Howieeeee/WorldScore and a leaderboard Space at huggingface.co/spaces/Howieeeee/WorldScore_Leaderboard. The dataset is downloaded by running python download.py, which places it under $DATA_PATH/WorldScore-Dataset.

## Sources

- [haoyi-duan/WorldScore on GitHub](https://github.com/haoyi-duan/WorldScore)
- [Issues](https://github.com/haoyi-duan/WorldScore/issues)
- [License: MIT](https://github.com/haoyi-duan/WorldScore/blob/main/LICENSE)
- [Project website](https://haoyi-duan.github.io/WorldScore/)
- [README](https://github.com/haoyi-duan/WorldScore/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/haoyi-duan-worldscore