# AlayaRenderer: inverting game frames into G-buffers and re-rendering them

> AlayaRenderer is a research release of two fine-tuned video diffusion models, one that decomposes RGB game footage into G-buffer channels and another that turns those channels plus a text prompt into a stylised video, together with a dataset of 4 million frames from two AAA games. The dataset and the data curation toolkit are still listed as pending, so the models and the paper are what you can use today.

**AlayaLab/AlayaRenderer** — Generative World Renderer: an AI-native Renderer for Games and Virtual Worlds.

- Repository: https://github.com/AlayaLab/AlayaRenderer
- Website: https://alaya-studio.github.io/renderer/
- Stars: 649 · Forks: 20
- Language: Python
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/alayalab-alayarenderer

## Two diffusion models that turn a game frame into buffers and back

The project presents a large-scale dataset and framework for inverse and forward rendering of videos using fine-tuned video diffusion models. The pipeline has two halves, and the direction of travel is the interesting part. The Inverse Renderer takes RGB video and decomposes it into G-buffer maps, namely albedo, normal, depth, roughness and metallic, which are the physical properties a renderer normally recovers from scene data. The Game Editing model goes the other way, taking G-buffer inputs plus a text prompt and producing a photorealistic RGB video with controllable lighting and style. The first half is a research tool for understanding what a frame is made of. The second is a production tool for changing what a frame looks like while keeping its structure. The authors, from a group that lists researchers including Zheng-Hui Huang, Zhixiang Wang, Jiaming Tan, Ruihan Yu, Yidan Zhang, Bo Zheng, Yu-Lun Liu, Yung-Yu Chuang and Kaipeng Zhang, published the paper in April 2026 and launched an online demo for the game editing side two days later.

## The Inverse Renderer is a fine-tune of a 7B diffusion renderer

The inverse model is fine-tuned from Cosmos-Transfer1-DiffusionRenderer, a 7B model from NVIDIA's Toronto AI Lab, and the fine-tuned weights are published on HuggingFace under the world_inverse_renderer repository. The usage note is specific about what a user has to do: follow the instructions in the inverse_renderer/ directory for environment setup and inference, download the weights, and replace the checkpoint that sits under inverse_renderer/checkpoints/Diffusion_Renderer_Inverse_Cosmos_7B with the fine-tuned one. That is the operational shape of the whole project. There is no packaged inference API and no configuration file to write. You assemble the upstream codebase, drop a checkpoint into a path that the upstream code expects, and run the upstream script. The G-buffer set it recovers, albedo, normal, depth, metallic and roughness, is the standard set a real-time renderer works with, which is why the inverse output can feed straight into the second model.

## Game Editing restyles G-buffers with a text prompt

The second model is fine-tuned from Wan2.1 1.3B through DiffSynth-Studio, and its weights live on HuggingFace under stylerenderer. Its input is the G-buffer set from the first half, and its instruction is a text prompt that controls lighting and style. The published example shows the whole call, with the checkpoint, the GPU index, a style name, a prompt, the directory holding the G-buffers and the output directory:

```bash
cd game_editing

CUDA_VISIBLE_DEVICES=0 python \
    examples/wanvideo/model_inference/inference_gbuffer_caption.py \
    --checkpoint models/train/Wan2.1-T2V-1.3B_gbuffer/model.safetensors \
    --gpu 0 \
    --style snowy_winter \
    --prompt "the scene is set in a frozen, snow-covered environment under cold, pale winter light with falling snowflakes, creating a silent and ethereal winter wonderland atmosphere." \
    --gbuffer_dir test_dataset \
    --save_dir outputs/ \
    --num_frames 81 --height 480 --width 832
```

The style argument and the prompt are two separate controls, and the example uses both, which is the point: a named style plus a natural language description of the scene you want. The frame count, height and width are passed on the command line rather than inferred, so the output geometry is set by the caller. The checkpoint is expected at models/train/Wan2.1-T2V-1.3B_gbuffer/ inside the game_editing directory, and the environment itself is set up by following the DiffSynth-Studio instructions, which the project points at instead of duplicating.

## Four million frames from two AAA games, at 720p and 30 FPS

The dataset is the more ambitious half of the release, and the numbers are specific. It holds 4M+ frames at 720p and 30 FPS with 6 synchronized channels, meaning the RGB image plus albedo, normal, depth, metallic and roughness. The footage covers 40 hours of gameplay from two AAA games, Cyberpunk 2077 and Black Myth: Wukong, extracted as synchronized RGB video and aligned G-buffer channels. The clips are long, averaging 8 minutes with continuous recordings up to 53 minutes, which matters for a diffusion model that has to keep temporal consistency over a long window rather than over a few frames. The content is varied: urban, outdoor and indoor scenes, and weather that includes sunny, rainy, foggy, night and sunset conditions. A motion blur variant is included, generated offline through sub-frame interpolation and linear-domain temporal averaging. Evaluation is handled by a VLM-based protocol that scores surface property predictions against real-world scenes without needing a reference, which is a practical answer to the problem that ground truth G-buffers for real footage do not exist.

## Cloning with submodules, and two environments to keep apart

The setup section is short, and the recommendation to use separate conda environments for the two models is the part to follow, because the project states it is to avoid version conflicts between the two codebases:

```bash
git clone --recurse-submodules https://github.com/ShandaAI/AlayaRenderer.git
cd AlayaRenderer
```

The clone command uses the recursive submodules flag, and the repository does carry a .gitmodules file, so a plain clone leaves you without the code the two model directories depend on. Note also that the clone command points at a GitHub organisation named ShandaAI, while the repository page for this project is under AlayaLab, and the documentation site is served from alaya-studio.github.io. Three names for one project is confusing on first contact, so it pays to confirm which one the checkpoints and instructions refer to. The weights themselves come from HuggingFace, one repository for the inverse renderer and one for the game editing model, and the base model weights they are fine-tuned from have to be present as well, since the fine-tuned checkpoint replaces a file inside the upstream project rather than standing on its own.

## Built on three upstream projects and published with a paper

The acknowledgements name the works this builds on: DiffusionRenderer from the NVIDIA Toronto AI Lab, Wan2.1 from Wan-Video, and DiffSynth-Studio from ModelScope. The project is a research artefact with a paper rather than a general purpose renderer, and the citation is given in BibTeX form, pointing at the Generative World Renderer article on arXiv with the author list and a 2026 year. The repository also carries translated READMEs, in Chinese, Japanese and Korean, and a docs/ directory alongside the assets, the two model directories and a .gitmodules file. The licence is Apache-2.0 according to the repository metadata, with the LICENSE file named in the README. The repository is not archived, and the last push was on 2026-05-05, which is the point to check if you are planning to depend on it, since there are no GitHub releases and no tagged versions to pin to.

## Two of the three promised artefacts are still open items

The TODO list at the end of the README is short and consequential. Releasing the dataset is unchecked, and so is releasing the data curation toolkit. That means the most interesting part of the project, the 4 million frame, six channel corpus, is described in detail but not yet downloadable, and the tooling that produced and curated it is not published either. What you can use now is the code for the two models and the paper behind them. Even for the models there is friction: the inference path is the upstream DiffSynth-Studio and Cosmos code with a replaced checkpoint, so a run needs the base weights, the fine-tuned weights and the right CUDA environment, and a mistake in any of the three paths shows up as a load error rather than a useful message. The example also fixes the output geometry on the command line, 81 frames at 480 by 832 in the published invocation, so a production pipeline would have to work out its own resolution and duration budget. This is a snapshot of a research release, and the repository's own list says what is still missing.

## Conclusion

AlayaRenderer is for research teams and technical artists who want to see what inverse rendering of game footage buys them, since the G-buffer decomposition and the prompt-driven restyling are both runnable with published checkpoints. It is not usable yet as a data source, because the dataset and the curation toolkit are still open items in the project plan, and inference assumes a CUDA GPU with a base model download in the right directory. Before starting, check the checkpoint paths in the two subdirectories against the HuggingFace model pages, because the models are fine-tunes that replace files rather than standalone packages.

## FAQ

### What does the AlayaRenderer Inverse Renderer produce?

It decomposes RGB video into G-buffer maps, namely albedo, normal, depth, roughness and metallic. The model is fine-tuned from Cosmos-Transfer1-DiffusionRenderer and its weights are published on HuggingFace.

### How do I run the AlayaRenderer game editing model?

Set up the environment by following the DiffSynth-Studio instructions, place the fine-tuned checkpoint under game_editing/models/train/Wan2.1-T2V-1.3B_gbuffer/, and run examples/wanvideo/model_inference/inference_gbuffer_caption.py with the checkpoint, GPU index, style, prompt, G-buffer directory and output directory as arguments.

### Is the AlayaRenderer dataset available yet?

Not yet. The README lists releasing the dataset and releasing the data curation toolkit as open items in its TODO list, so only the model code, the weights and the paper are available at the moment.

### How large is the AlayaRenderer dataset?

It is described as 4M+ frames at 720p and 30 FPS with 6 synchronized channels, covering 40 hours of gameplay from two AAA games, Cyberpunk 2077 and Black Myth: Wukong, with clips averaging 8 minutes and reaching 53 minutes of continuous recording.

### Do I need separate environments for the two AlayaRenderer models?

The usage section recommends creating separate conda environments for the two models, explicitly to avoid version conflicts between the upstream codebases they are built on.

## Sources

- [AlayaLab/AlayaRenderer on GitHub](https://github.com/AlayaLab/AlayaRenderer)
- [Issues](https://github.com/AlayaLab/AlayaRenderer/issues)
- [License: Apache-2.0](https://github.com/AlayaLab/AlayaRenderer/blob/master/LICENSE)
- [Project website](https://alaya-studio.github.io/renderer/)
- [README](https://github.com/AlayaLab/AlayaRenderer/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/alayalab-alayarenderer