Pixal3D: pixel-aligned 3D generation from a single image

What Pixal3D does that attention-based image-to-3D does not

Most image-to-3D systems condition a generative model on image features passed through attention layers. The image guides the output, but nothing forces a given pixel to land on a given surface point. Pixal3D takes the other route. The README states that it "explicitly lifts pixel features into 3D through back-projection, establishing direct pixel-to-3D correspondences." The stated payoff is near-reconstruction-level fidelity with detailed geometry and PBR textures.

The audience is narrow and technical. You need a working TRELLIS.2 environment, a CUDA toolchain capable of compiling natten, and enough VRAM for a 1536-resolution cascade. This is a research release from Tsinghua University (BNRist), Tencent ARC Lab and Victoria University of Wellington, published at SIGGRAPH 2026, not a desktop application. The repository ships inference.py, inference_mv.py, train.py, a data_toolkit directory and a Gradio app.py. If you only want to see the output, the README points at a Hugging Face Space and you can skip the install entirely.

Back-projection, the cascade, and what the two branches contain

The mechanism is easiest to describe as a constraint rather than a feature. Instead of asking the network to infer where a pixel belongs, the pipeline projects pixel features into 3D space so the correspondence is defined before generation. The README contrasts this directly with methods that "loosely inject image features via attention."

The repository carries two branches, and picking the wrong one changes what you get. The main branch is described as the latest version, rebuilt on the TRELLIS.2 backbone with better performance. The paper branch is the original implementation on Direct3D-S2, corresponding to the numbers in the SIGGRAPH 2026 paper. The README is explicit: if you want to reproduce the paper results, switch to the paper branch. Anyone comparing Pixal3D against published tables without doing that checkout is comparing against a different codebase.

Multi-view inference reuses the same cascade but conditions it on several views at once, and it loads a separate set of weights from ckpts/*_mv selected by pipeline_mv.json, drawn from the same Hugging Face repository. That is a real architectural split, not a flag on the single-image path: different weights, different pipeline config, different entry point.

Installing Pixal3D on top of TRELLIS.2

There is no standalone installer. Step one in the README is to follow the TRELLIS.2 installation guide and set up that base environment first. Everything below assumes you have done that and are inside it.

Step two pulls the project's own pinned dependencies, which include diffusers, transformers, trimesh, kornia and a MoGe entry installed straight from GitHub.

pip install -r requirements.txt

Step three is the part that fails most often. natten is a CUDA extension, and the README's command requires you to substitute your own CUDA architecture and worker count for the xx placeholders.

NATTEN_CUDA_ARCH="xx" NATTEN_N_WORKERS=xx pip install natten==0.21.0 --no-build-isolation

Step four installs a prebuilt utils3d wheel hosted in the author's own release storage.

pip install https://github.com/LDYang694/Storages/releases/download/20260430/utils3d-0.0.2-py3-none-any.whl

The README notes that requirements-hfdemo.txt is for the Hugging Face Spaces demo on H-series GPU architecture and may not be compatible elsewhere. Do not install it on a workstation and expect it to behave like requirements.txt.

Your first GLB, and the VRAM and attention switches

The single-image entry point takes an image path and an output path. The README's example writes a GLB mesh.

python inference.py --image assets/images/0_img.png --output ./output.glb

If the default run exhausts memory, --low_vram loads models on demand rather than holding them all resident. The trade is resolution: the pipeline defaults to 1536 in standard mode and 1024 in low-VRAM mode, and --resolution overrides either default in either direction.

python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram --resolution 1536

One more switch matters if you never built flash_attn. The README says you can fall back to PyTorch's built-in SDPA backend by setting the ATTN_BACKEND environment variable to sdpa on the command.

ATTN_BACKEND=sdpa python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram

For several views of one object, inference_mv.py points at a directory instead of a file. It accepts --low_vram, --resolution and ATTN_BACKEND with the same behaviour, plus --num_views N to restrict the run to the first N views.

python inference_mv.py --views_dir assets/mv_images/example --output ./output_mv.glb

The multi-view input contract is stricter than it looks

The views directory needs a transforms.json alongside the images, using the Blender/NeRF camera convention: transform_matrix is a 4x4 camera-to-world matrix, camera_angle_x is the horizontal FOV in radians, the world is Z-up, and each camera looks along its own negative Z with its own positive Y as up. camera_angle_x can also be set per frame, overriding the top-level value.

Two details in the README are easy to miss and expensive to discover later. Views are never cropped or rescaled, so the framing you supply has to be the framing the cameras describe; a tightly cropped object in a loosely specified camera rig will not be corrected for you. And the first frame is treated as the main view, which should be the canonical front view, with the camera at (0, -d, 0) looking at the origin.

Alpha channels are optional. When present, alpha is taken as the object mask; when absent, the view is segmented automatically with the same background-removal model inference.py uses. If your views are the ordinary four-shot orbit at eye level, the README says you can reuse the shipped assets/mv_images/example/transforms.json unchanged and just repoint each file_path. That file encodes azimuths 0, 90, 180 and 270 degrees, elevation 0 and a 20 degree FOV, which is a useful sanity check on whether your capture matches the assumption.

Where Pixal3D is the wrong tool, and what to use instead

The clearest failure mode is the camera rig. Pixal3D's multi-view path assumes you know where the cameras were, or that your capture matches the shipped orbit preset. If you have a handful of photos from unknown positions, the transforms.json contract is a wall, not a detail. A feed-forward reconstruction tool that estimates poses from images is the better fit there, because it solves the problem Pixal3D asks you to have already solved.

The second boundary is the base environment. Pixal3D is not self-contained; it is an addition to TRELLIS.2. If you have already built TRELLIS.2, the incremental cost is a requirements install, a natten build and a wheel. If you have not, you are adopting TRELLIS.2 first and Pixal3D second, and the natten build is the step most likely to consume an afternoon on an unusual CUDA architecture.

Compared with TRELLIS.2 itself, the difference is the conditioning mechanism rather than the backbone. The main branch shares TRELLIS.2's architecture, so the comparison is about how the image enters the model: back-projected pixel features with defined correspondences, versus attention-based feature injection. Compared with the paper branch's Direct3D-S2 base, the difference is the whole implementation, which is why the README separates reproduction from the current release. The search phrase pixal3d vs trellis is really asking about that first comparison, and the README answers it in one line about back-projection.

Licence, maintenance and what upgrading costs you

The repository is MIT licensed, and a separate NOTICE file sits at the top level alongside LICENSE. The practical implication is that the base environment is not covered by that MIT grant in the same way: TRELLIS.2 is a Microsoft project and its own terms apply to the code you install in step one, and the requirements file pulls MoGe directly from GitHub. Read both before shipping anything. This is a description of the files present, not legal advice.

The last push to the default branch was on 2026-09-01, and the repository is not archived. The news list gives the release cadence: inference code and an online demo in May 2026, a TRELLIS.2-based improvement the same month, training code and a data preparation toolkit in May 2026, and multi-view inference code in September 2026. That is a project still adding surface area rather than one in maintenance mode.

Upgrade cost is dominated by the split between main and paper. Improvements land on main; paper fidelity lives on paper. If you need both, you maintain two checkouts and two environments, and the natten and utils3d steps are the ones worth scripting. There are no retrieved releases, so pinning is done through requirements.txt and the branch you check out, not through versioned tags.

Editorial conclusion

Adopt Pixal3D if you already run TRELLIS.2 or Direct3D-S2 and want image-consistent geometry and PBR textures from a single photo, or if you have a few orbit shots and want the multi-view path. Do not adopt it if you cannot build natten against your CUDA architecture, if you need a paper-exact reproduction without checking out the paper branch, or if you expected a ComfyUI node, a GGUF build or a standalone installer, none of which the repository describes. Before committing, verify the paper branch for reproduction, confirm the default 1536 resolution fits your GPU or plan on --low_vram, and read NOTICE alongside the MIT LICENSE because the base environment comes from TRELLIS.2.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. TencentARC/Pixal3D on GitHub