PixelDiT and the argument for generating in pixel space

Dropping the VAE from the diffusion pipeline

PixelDiT is a series of works on pixel-space generative modeling, and the stated aim is to remove the need for the VAEs that traditional latent generative models rely on, performing generation directly in pixel space.

That is a bigger change than it sounds. In a latent diffusion model, a VAE compresses an image into a low dimensional latent space, the diffusion model learns that latent distribution, and a second decoder maps samples back to pixels. PixelDiT removes that pair of stages and runs the diffusion transformer over pixels. The reason to try it is that the VAE is a lossy bottleneck: information destroyed by the encoder cannot come back, and the encoder and decoder are extra training runs and extra inference cost. The reason to be cautious is that a transformer operating on a 256x256 image has far more tokens than one operating on a 32x32 latent, which is where the compute argument cuts the other way.

The work comes out of NVIDIA and the University of Rochester, with Yongsheng Yu, Wei Xiong, Weili Nie, Yichen Sheng, Shiqiu Liu and Jiebo Luo listed as authors, Wei Xiong as project lead and main advisor, and Jiebo Luo of Rochester among them.

Two generations of model are in the repository. PixelDiT is the first, released as a CVPR 2026 Oral and named a CVPR 2026 Best Paper Finalist, with ImageNet models and a 1.3B parameter text-to-image model that generates 1024x1024 images. PixelDiT2 is the follow-up, accepted to NeurIPS 2026, which grounds generation in learned representations and, according to the README, converges significantly faster and obtains better results on ImageNet.

PixelDiT2 grounds generation in a DINOv3 encoder

The mechanism that distinguishes PixelDiT2 from PixelDiT is representation grounding. The README describes the second generation as a deeper investigation into how representation learning can guide content generation for pixel diffusion, and the project proposes using representations to ground the generation process.

The implementation of that idea is visible in the packaging metadata rather than in the README prose. The optional dependency group named pixeldit2 pulls in timm, and the comment in pyproject.toml explains why: the grounding encoder in pixdit_core.grounding creates its DINOv3 backbone with timm. In other words, the grounding signal is a self-supervised vision backbone, and it is a separate encoder path rather than a modification of the denoiser itself.

The package itself is small in structure. The project name is pixeldit, the version is 0.1.0, and the description reads PixelDiT and PixelDiT2 model definitions. The setuptools configuration packages only pixdit_core, and the base dependency list is just torch and numpy. Everything else sits in the surrounding directories, c2i/ for class to image, t2i/ for text to image, and tools/ at the root, with the requirements files handling the heavier environment.

The consequence of that layout is that the model definitions are a library, while the training and sampling scripts are repository-local programs. Anyone who wants to use a checkpoint inside a larger system can depend on pixdit_core and supply their own loop, and the inference guides under c2i/README.md and t2i/README.md describe how the reference scripts drive it.

Two requirement files, two evaluators, two environments

The installation instructions are short because the environments are complicated. The README asks you to install the requirements for the model family you want to use, in separate Python environments, and gives two commands for exactly that:

python -m pip install -r requirements-pixeldit2.txt

python -m pip install -r requirements-pixeldit1.txt

The separation is not cosmetic. PixelDiT2 uses Python 3.10 with pinned PyTorch and CUDA dependencies, while the PixelDiT line keeps its original dependency specification. The top-level requirements.txt is retained only as a backward compatible entrypoint that points at the first generation:

# Backward-compatible entrypoint for the original PixelDiT.
# For PixelDiT2, install requirements-pixeldit2.txt instead.
-r requirements-pixeldit1.txt

The evaluation environments differ too, and this is the detail that most affects reproducing numbers. The README states that C2I evaluation differs: PixelDiT2 uses the LTH14 torch-fidelity fork with JiT statistics, while PixelDiT uses ADM in a separate evaluation environment. FID numbers from different evaluators are not comparable, so a comparison between the two rows of the results table has to be made inside one environment.

For the grounding extra itself, the packaging keeps it optional:

[project.optional-dependencies]
# PixelDiT2's grounding encoder (pixdit_core.grounding) creates its DINOv3 backbone with timm.
pixeldit2 = ["timm"]

Both model families share the same class to image entry point, c2i/main.py, so the choice between generations is a configuration and checkpoint decision rather than a different program to learn.

What the ImageNet numbers are made of

The class to image results are given for ImageNet at two resolutions, with FID and inception score:

| Model | Resolution | Epochs | FID | IS | |---|:---:|---:|---:|---:| | PixelDiT2-H/16 | 256x256 | 600 | 1.46 | 301.6 | | PixelDiT2-H/16 | 512x512 | 680 | 1.48 | 295.7 | | PixelDiT-XL | 256x256 | 320 | 1.61 | - | | PixelDiT-XL | 512x512 | 850 | 1.81 | - |

The numbers only mean something with the sampling configuration attached, and the README supplies it. Results use 50,000 samples. PixelDiT2 uses Heun with 50 steps, while PixelDiT uses FlowDPMSolver with 100 steps. So PixelDiT2 is sampled with roughly half the steps of its predecessor, and the table is not comparing equal compute per image.

There is also a training caveat that the README states plainly, and it is unusual enough to matter for reproduction. The models are resumed every 4 hours, using the timestamp as the random seed each time, and the final training outcome may have a slight gap compared to a continuous run without intermediate resumes. Anyone trying to hit FID 1.46 by training from scratch should know that the released numbers come from a run that was interrupted roughly every four hours, with the timestamp as seed, and that the epoch counts in the table are part of the recipe.

The class to image guide lists all checkpoints and the corresponding sampling commands, and the model list in that guide is where a matching checkpoint and configuration have to be chosen before inference.

Text to image at 1024, and the ComfyUI shortcut

The text to image model is PixelDiT-T2I, the 1.3B parameter model, evaluated with GenEval and DPG-Bench:

| Resolution | GenEval | DPG-Bench | |:---:|---:|---:| | 512x512 | 0.78 | 83.7 | | 1024x1024 | 0.74 | 83.5 |

The text to image guide under t2i/README.md covers training and inference for this model, separate from the class to image path, which is why the two model families can share c2i/main.py without sharing their documentation.

The most practically useful part of the README for people who do not want to train anything is the ComfyUI route. The README states that PixelDiT-T2I is also available in ComfyUI, and that the Comfy-Org/PixelDiT repository provides repackaged weights in bf16 and mxfp8 along with a ready to use text to image workflow template. The repackaged bf16 and mxfp8 formats matter for hardware selection, since they are the two precision paths on offer and they decide what fits on a given card.

The README also points to PiD, a separate project hosted under the nv-tlabs organisation.

On the Hugging Face side, the model repositories are nvidia/PixelDiT-ImageNet, nvidia/PixelDiT2-ImageNet and nvidia/PixelDiT-1300M-1024px, and the project pages are at pixeldit.github.io for PixelDiT and pixeldit.github.io/pixeldit2 for PixelDiT2, with the papers on arXiv at 2511.20645 and 2609.24919.

Post-modulation for PiT blocks, and the gaps in the docs

One training detail is called out in the news section because it is a change to the model, not a new checkpoint. In June 2026 the project added post-modulation for PiT blocks to improve training stability, with a dedicated section in the c2i guide. PiT blocks are the transformer units the model is built from, so this is an architectural addition with a stability rationale, and it means a checkpoint trained before that change and one trained after are not simply two points on the same curve.

The timeline in the README is a useful map of the project's shape. The paper was released in November 2025, PixelDiT was accepted to CVPR 2026 as an Oral in February 2026, training and inference code with pretrained models were released in April 2026, the Best Paper Finalist news and the post-modulation change came in June 2026, and PixelDiT2's NeurIPS 2026 acceptance landed in September 2026.

The documentation gaps are real. The README does not show the actual inference command line for c2i/main.py, only that both families use it, so the flags for a checkpoint or a sampling config have to be read out of the c2i guide. The README does not state a minimum CUDA version, a GPU memory requirement, or the training hardware for the released runs. It also does not document how the VAE-free approach interacts with downstream edit pipelines, since there is no latent space to edit.

The repository layout carries a naming inconsistency worth noting: the package directory is pixdit_core, with a single d, while the project is spelled PixelDiT throughout. Anyone scripting imports has to match the directory name, not the brand name.

A custom licence, no releases, a busy commit history

The licence is the first thing to check and it is not a standard one. The repository is published under a custom licence that GitHub cannot classify, with the text in the LICENSE file and the package metadata pointing at both LICENSE and NOTICE. The packaging configuration also lists requires-python as 3.10 or above and marks the project version as 0.1.0, which for a paper repository is a fair signal of the packaging maturity, since the version tracks the model definitions rather than a release cadence.

This is not legal advice, but the practical boundary is straightforward. Anyone planning to use these checkpoints in a product, or to ship them inside a hosted service, has to read LICENSE and NOTICE themselves, because a custom licence can carry conditions that a permissive one would not.

On maintenance, the repository is not archived and the last push was on 2026-10-16. The project has no GitHub releases at all, so there is no tag to pin and no published changelog of release notes. The news section in the README and the git history are the only version record, and the version in pyproject.toml stays at 0.1.0 regardless of which model generation you use.

For a research codebase, that is a workable setup, especially with checkpoints on Hugging Face carrying their own revision history and the Comfy-Org repackaging giving a second, more convenient distribution. For a production dependency, the absence of tags means you track the branch yourself, and the post-modulation change in June 2026 is a reminder that the model definition can move under you.

Editorial conclusion

PixelDiT fits a research group that wants a diffusion transformer without a latent autoencoder, or that wants to study how representation grounding changes convergence, and it does not fit a team that needs a permissive licence or a tagged release to pin. Before adopting it, read LICENSE and NOTICE, then reproduce the ImageNet numbers with the sampling settings in c2i/README.md, because the reported FID depends on the solver and the step count.

Frequently asked questions

Official sources

1. Issues
2. NVlabs/PixelDiT on GitHub
3. README