AlayaWorld: an open, playable video world model with long-horizon memory

A world model you can actually play

AlayaWorld, from Alaya Lab, is an interactive autoregressive world model: it generates a video world continuously while you act inside it. Two control channels make it a toy you steer rather than a clip you watch. A rendered 3D cache with lightweight AdaLN camera modulation gives camera movement grounded in an actual trajectory, and chunk-level prompt switching lets you introduce new events mid-generation at chunk boundaries.

Since 2026-08-20 there has been a browser demo in the reactor directory that streams the generation live, driven from the keyboard. The intended audience is visible in the release notes: world-model researchers, people building interactive experiences, and anyone who wants to inspect a full training and inference stack rather than consume an API. The project carries three arXiv reports, an intro, a full report and a v1.1 report.

Four properties, each with its own mechanism

The README organizes the design as four properties, and each maps to a concrete mechanism. Interaction comes from the two channels above. Consistency comes from complementary memory: an explicit 3D cache reprojected into the queried view for spatial recall, plus a compressed frame-history embedding for temporal continuity, so a place you revisit stays recognizable.

Stability over long rollouts is engineered directly: the model trains on drifted histories, and an error bank re-injects accumulated artifacts into both memory and target, so errors get seen during training instead of compounding unseen. Runtime comes from few-step DMD distillation and short temporal chunks, which is what makes real-time interaction possible while keeping prompt switches at chunk boundaries to cap latency.

Quick start: from pins to a one-minute video

One Python environment covers training and all inference paths, verified on torch 2.7.1 with CUDA 12.8. The README's environment step:

pip install -r requirements.txt
git clone https://github.com/ByteDance-Seed/Depth-Anything-3 third_party/Depth-Anything-3
pip install -e third_party/Depth-Anything-3

The extra clone exists because the da3 inference path depends on Depth-Anything-3 as a code repository installed after the pinned packages. Inference has one launch form selected by config:

# a) case demo (da3 spatial memory): first-frame image + camera.pt + prompt -> ~1 min video
CONFIG_PATH=configs/infer.yaml bash scripts/finetune/train.sh
bash inference/run.sh
python -m inference.run --input playground/case1/case1 --seed 1234 --rounds 45

The shortcut run uses the bundled playground case, so the first successful generation needs no data of your own. Weight locations live under paths in each config and should be repointed to wherever your downloads sit.

The weight manifest is the real bill of materials

Running AlayaWorld means assembling pieces from several organizations, and the README's weights table is honest about it. The da3 path needs merged_infer.safetensors, a bundle of DiT, VAE, text encoder and history encoder from AlayaLab. Training and AR/DMD inference need the LTX-2.3 base from Lightricks, plus AlayaLab's v1.1 stage2b AR teacher and stage3 few-step student. Everything needs a Gemma text encoder from Google, which is gated, so a Hugging Face account with access approval is part of the install.

The training path adds a ViGeo checkpoint from a third party and the da3 path needs Depth-Anything-3 weights. Count the dependencies before budgeting disk, and note what the table implies: this stack leans on LTX-2.3 as its generative backbone, with AlayaWorld's contribution being the memory, control and stability machinery wrapped around it.

The licence needs reading before you build on it

GitHub's licence detection reports NOASSERTION for this repository, which signals a custom or non-standard licence rather than MIT or Apache. A NOTICE file and THIRD_PARTY_LICENSES.md sit alongside LICENSE, consistent with a stack that bundles third-party components with their own terms.

The practical instruction: read LICENSE and THIRD_PARTY_LICENSES.md before any commercial or hosted use, and do not assume the code being public means it is permissive. Everything else about openness is generous. The release roadmap is fully checked: inference code, pretrained weights, v1.1 AR and DMD weights, training code, and partial training data as a Hugging Face dataset, with docs/vigeo serving as a full training handbook covering data format, stages and knobs.

Where the stack will strain

The bill of materials tells you the hardware story before any benchmark does: the base is an LTX-2.3 22B-class dev model, and the README verifies only torch 2.7.1 plus CUDA 12.8, without publishing a VRAM figure. Assume serious GPUs for training, and expect the DMD few-step student to be the realistic path to interactive rates on anything smaller.

Three further frictions are visible from the layout. The three inference paths, da3, AR and DMD, share one launcher but differ in weights and knobs, so the first hour is config archaeology. The gated Gemma encoder adds an approval wait to day one. And the training handbook's existence, thorough as it is, confirms that reproducing training is a project, not an afternoon.

Against passive generators and game engines

Two existing answers bracket this project. Video generation models produce footage and stop: no camera you control, no prompt mid-roll, no memory of where you have been. Game engines give real rules and total control but no learned generation: every rock is placed by hand or procedure, and nothing is imagined.

AlayaWorld sits deliberately between. Generation is learned and continuous like a video model, but interaction, camera control, mid-roll event injection and spatial memory make it behave like a place rather than a clip. The cost of that position is the complexity documented above: a multi-org weight manifest, a custom licence, and hardware demands that neither a plain video generator nor a game engine would impose. For researchers who want to poke at that middle ground with code in hand, the stack is now inspectable end to end.

Editorial conclusion

AlayaWorld is worth assembling if you research interactive world models and can absorb a gated text encoder, multi-org weights and serious GPUs; the full stack from training code to a browser demo is public and the v1.1 report explains the design. Skip it if you want a permissively licensed component, a documented VRAM budget, or a plug-and-play demo beyond the bundled case. Before anything else, read LICENSE and THIRD_PARTY_LICENSES.md, since the NOASSERTION flag means the terms are custom and they govern everything you might build on top.

Frequently asked questions

Official sources

1. AlayaLab/AlayaWorld on GitHub
2. Issues
3. README