PiD: NVIDIA's pixel diffusion decoder that merges decoding and upscaling

One module where two used to be

Every latent diffusion pipeline ends the same way: a VAE or RAE decoder turns the latent into pixels, deterministic, fast, and bounded by the resolution it was trained on, with upscaling handled afterwards by a separate model if it is wanted at all. PiD collapses that into a single generative module. It reformulates the latent-to-pixel decoder as a conditional pixel-space diffusion model, denoising directly at high resolution and producing a super-resolved image in one pass.

The framing is a decoder swap rather than a new pipeline: plug-and-play, replacing the VAE decoder, sitting after whatever latent diffusion backbone produced the representation. That positioning is why it integrates into existing workflows instead of demanding new ones.

The pedigree is research-grade: an NVIDIA labs project page, a paper at arXiv:2605.23902, model weights under the nvidia organization on Hugging Face, and an author list spanning NVIDIA and University of Toronto collaborators. The licence needs the usual caveat for this publisher: GitHub reports no standard licence, so the repository's own licence terms govern whatever the README does not.

The backbone matrix and the timeline

The checkpoint catalogue is the practical story. Initial release in late May 2026 shipped PiD options for FLUX, FLUX.2, Z-Image, Z-Image-Turbo, SD3, DINOv2 and SigLIP. Two weeks later, SDXL, Qwen-Image and a 2512-resolution Qwen-Image variant joined, alongside a codebase cleanup and a torch.compile mode.

In July, the training code landed with PixelDiT and PiD v1.5 checkpoints for two-thousand-to-four-thousand upscaling, in both distilled and undistilled forms, with v1.5 variants covering FLUX including Z-Image and Z-Image-Turbo, FLUX.2 and Qwen-Image. Days after that, optional text-to-image support arrived through Boogu-Image, both native generation and PiD decoding from that model's Flux-style VAE latents.

Read as a timeline, the project shipped decode, then breadth of backbones, then training, then generation integration, in roughly two months. The ComfyUI merge came early, at the end of May, which says the authors understood where adoption for this kind of module actually happens.

Install, environment, and a verification script

Two install paths are documented. The quick start reuses an existing environment that already has PyTorch with CUDA, transformers and diffusers, adding only the utility packages the inference code imports, a list the README gives as a single pip invocation. The from-scratch path uses uv:

uv python install 3.12
uv sync --frozen
source .venv/bin/activate
PYTHONPATH=. python verify_env.py

That sequence pins Python 3.12, syncs locked dependencies into a project-local virtual environment, and then runs a verification script that prints an explicit pass message only after checking all required imports and CUDA. Publishing an environment validator, and gating the docs on its output, is a small courtesy that saves the most common first-hour failure in this genre.

Commands run from the repository root with the source tree on the Python path, a constraint the quick start states plainly. Checkpoints then pull directly from the model hub:

hf download nvidia/PiD --local-dir . --include "checkpoints/*"

A justfile in the repository wires pre-commit hooks for development, with lint and formatting aliases, which sketches the contributor workflow without documenting it further in the README's visible sections.

Two entry points, three checkpoint variants

Inference exposes two scripts, and their names describe the two ways a decoder gets used. The first takes text or class input through a latent diffusion backbone and then decodes with PiD: generation end to end. The second takes an existing image, encodes it with a VAE, and decodes with PiD: pure super-resolution of real images. Both select the backbone with a flag.

The checkpoint variant flag is the knob that matters for quality and cost: a default two-thousand-resolution decoder, a two-to-four-thousand upscaling variant, and the v1.5 generation of that upscaler, with the distilled and undistilled forms trading inference steps against fidelity in the usual diffusion bargain.

What the entry-point design implies is worth stating: because decode is a pure function of the latent, PiD composes with any pipeline that can hand it one, which is exactly the property that made it integrable into ComfyUI as a node rather than a fork.

In ComfyUI, where the users are

Support was merged into ComfyUI itself at the end of May 2026, per the repository's news section citing the pull request. That single fact changes the adoption calculus more than any benchmark in the paper: the audience for upscaling already lives in node graphs, and a decoder that drops in as a node reaches them without a repository checkout, a Python environment, or any of the installation path above.

The standalone repository remains the reference implementation and the only place to train or fine-tune, since the merge covers inference. For workflow builders, the practical questions become the usual ones: which backbone's latents are being decoded, which checkpoint variant matches the target resolution, and whether the generative character of the decode is desirable for the content at hand.

That last point is the honest double edge. A diffusion decoder invents detail a deterministic decoder would not, which reads as restoration on some content and as fabrication on others, and any pipeline adopting it inherits that judgement call.

Costs, limits, and the two-stage alternative

The limits are physical and economic. Denoising in high-resolution pixel space is expensive relative to a one-shot VAE pass; the distilled checkpoints exist precisely to cut that cost, and the torch.compile mode exists for the same reason. Coverage is per-backbone, so an unsupported model needs either a training run or a different decoder. And the licence is the publisher's custom terms rather than a standard open-source grant, which commercial adopters should read before shipping.

The alternative is the incumbent arrangement the README positions against: a standard VAE decode followed by a dedicated upscaler, convolution-based super-resolution or a tiled diffusion upscaler. It is cheaper at the deterministic end, composes with everything, and adds no generative surprises; it also caps at what the upscaler was trained to hallucinate and routes through two models instead of one learned module.

PiD's bet is that decoding and upscaling want to be one generative act, learned jointly. For workflows where that bet pays, the checkpoint matrix, the training code and the ComfyUI integration make it testable this afternoon, which is the correct standard for a research artifact aspiring to tool status. The last push was on 2026-07-22.

Editorial conclusion

PiD fits diffusion pipeline builders and ComfyUI users who want decoding and super-resolution as one generative step, with released checkpoints spanning FLUX, FLUX.2, SD3, SDXL and Qwen-Image variants, plus training code for new backbones. It does not fit latency-sensitive or deterministic-decode requirements, since pixel-space diffusion costs more and invents detail, or anyone needing a standard open-source licence, as the repository carries custom NVIDIA terms. Verify first: that a checkpoint variant exists for your backbone and target resolution, whether the distilled v1.5 upscaler's speed and quality trade suits your volume, and the licence terms for commercial use. The last push was on 2026-07-22.

Frequently asked questions

Official sources

1. Issues
2. nv-tlabs/PiD on GitHub
3. Project website
4. README