RenderFormer renders a triangle mesh with global illumination

Rendering as sequence to sequence

The framing is the interesting part of the paper, and the repository explains it in one paragraph. RenderFormer is a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects, and it does not require per-scene training or fine-tuning. Instead of taking a physics-centric approach, the project formulates rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted into a sequence of output tokens representing small patches of pixels. So the model is not predicting a colour per triangle and shading it, it is producing pixels directly from a tokenised description of the geometry. The gallery in the project page shows triangle-mesh scenes rendered with multiple specular reflections, complex shadows, diffuse indirect lighting, glossy reflections, soft and hard shadows and multiple light sources, all without per-scene training or fine-tuning.

A view-independent stage and a view-dependent one

The pipeline has two stages and the split is the architectural idea. The first is a view-independent stage that models triangle-to-triangle light transport, which is the part of the computation that does not depend on where the camera is, since the bounce of light from one surface to another is a property of the scene. The second is a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values, guided by the triangle sequence coming out of the view-independent stage. So the first stage works out how light moves between surfaces and the second stage turns a camera's rays into colours. Both stages are based on the transformer architecture and are learned with minimal prior constraints, which is the phrase that distinguishes this from a learned renderer with a physics model baked in. The evaluation described in the paper is on scenes with varying complexity in shape and light transport.

No per-scene training is the whole pitch

Every other learned renderer in this space tends to need a training step per scene, an optimisation that fits the model to the geometry and lighting in front of it. RenderFormer is built so that it does not, and the example gallery makes the claim concrete by showing scenes that have never been trained on. For a user this is the difference between a system you can point at a new mesh and a system that needs a GPU hour per asset. The price of that property is not stated as such in the repository, but it is inherent in the approach: the model has learned a general mapping from triangle tokens to pixel tokens, so its output is a learned approximation of the physics rather than a physical solution, and the renderings in the gallery are presented as examples of what it produces rather than as ground truth. The examples directory makes the range of test scenes concrete, with configs for a box, a box and bunny, a box and lucy, a box and teapot, a compose scene, a constant width case, crystals, a fox in the wild, a horse and heart, a renderformer logo, a room, a shader ball, a tree and the Veach MIS test case.

Two pretrained models, and a strict weight format

Model loading has a rule that is worth reading before you plan a deployment. Model loading accepts only model.safetensors weights, for both Hugging Face repositories and local directories, and both official models already use that format. Custom models should provide config.json and model.safetensors, as written by RenderFormer.save_pretrained(), and missing or invalid safetensors files cause loading to fail, with pickle weights such as pytorch_model.bin never used as a fallback. Two models are published. RenderFormer-V1-Base has 205M parameters and the model id microsoft/renderformer-v1-base, and RenderFormer-V1.1-Large has 483M parameters with the id microsoft/renderformer-v1.1-swin-large, and the naming of the latter points at a Swin backbone. The safetensors rule is a deliberate security posture for a repository that loads weights from the internet, and it also means you cannot feed it a checkpoint from a generic PyTorch training run without converting it first.

From a scene JSON to an image

The usage path is a scene conversion step followed by an inference step, and the example scenes are JSON files in the examples directory. The first command turns a scene config into the HDF5 scene format the pipeline reads:

python3 scene_processor/convert_scene.py examples/cbox.json --output_h5_path tmp/cbox/cbox.h5

The second runs the model and writes the result:

python3 infer.py --h5_file tmp/cbox/cbox.h5 --output_dir output/cbox/

The output folder then contains an EXR file and a PNG file for the view. The EXR is the HDR linear output from RenderFormer and the PNG is the LDR version of the rendered image, and the tone mapper can be changed with the tone_mapper option to get better visual results. There is a batch script for many scenes, shell scripts for rendering images and videos, a script that downloads the example video data, and a tests directory. For your own scene the documentation covers a scene definition JSON with the object, camera and scene structure, the HDF5 data fields, a remeshing step for objects, and rendering a reference image with Blender so you have something to compare the model against.

What it needs to run

The hardware and environment requirements are modest in description but firm in practice. The code is tested on Linux, MacOS and Windows, and on both NVIDIA CUDA GPUs and Apple Metal GPUs, with a minimal GPU memory requirement of 8GB. The environment needs PyTorch 2.0 or newer, and for CUDA users the documentation suggests installing Flash Attention from the Dao-AILab repository. The rest of the dependencies are installed from the repository, and one of them deserves a note, because it is a direct Git reference rather than a released package:

dacite
einops
roma
huggingface_hub>=0.23.5
safetensors>=0.4.3
trimesh
scipy
pymeshlab
h5py
natsort
imageio[ffmpeg]
simple-ocio

The scene conversion also needs the freeimage plugin for imageio, which the setup command downloads explicitly because it is needed for HDR image IO. The attention backend is negotiable. The script automatically falls back to torch scaled dot product attention if Flash Attention is not found on the system, and an environment variable lets you choose the implementation deliberately, which matters if you are comparing runtimes or debugging a numerical difference between a CUDA machine and a Metal one.

The released models are not the models in the paper

The most important line for anyone reproducing the results is in a note on the released models. The authors found a shader bug in the training data that they used in the submission, they re-trained the models with the corrected shader, and they released the new models, so the model performance and output might be different from the results in the paper. That is the right thing to do and it is also a warning: a table of numbers from the SIGGRAPH paper will not match what the released checkpoints produce on the same scene, so do not treat a mismatch as a bug in your setup without reading this note first. The project also publishes the paper PDF, the arXiv entry, the project page and a citation file, and the authors are Chong Zeng, Yue Dong, Pieter Peers, Hongzhi Wu and Xin Tong. The repository is MIT licensed, is not archived, and the last push was on 2026-10-13, with no GitHub releases published, so the model identifiers on Hugging Face are the thing to pin.

Editorial conclusion

RenderFormer fits a graphics researcher who wants to see what a learned renderer does with a mesh it has never seen before, since the absence of per-scene training is the whole claim and the example scenes cover boxes, rooms, teapots, foxes and the classic test objects. It does not fit a production renderer that needs predictable, physically based results on demand, since the output is a learned approximation, and it does not fit a machine without a GPU, because the minimum GPU memory requirement is 8GB and the pipeline expects PyTorch 2.0 or newer. Before you rely on it, read the note about the released models, since a shader bug in the training data was found and the models were re-trained, so the output differs from the paper, and check the attention backend, because the script falls back to torch scaled dot product attention when Flash Attention is not found.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. microsoft/renderformer on GitHub
4. Project website
5. README