What is a Diffusion model?

How a diffusion model generates an image

The idea is a fixed corruption process with a learned inverse. Take an image, add a little Gaussian noise, add a little more, and keep going until what remains is pure noise. That forward process has no parameters and no choices in it. Training consists of learning a network that, given a noisy image and the step index, predicts the noise that was added. The most common parameterisation predicts the noise itself; variants predict the direction of change or the score of a noise distribution, and the choice changes the loss but not the shape of the pipeline.

Generation is the process run backwards. Start from random noise, ask the network what noise it thinks is there, remove a little of it, repeat. The network is a U-Net shaped convolutional architecture with skip connections, because the resolution and the number of channels change as you go down and back up the path.

Two additions make it useful rather than merely possible. Conditioning means the network is told what to make, and in text-to-image models it is told by embedding the prompt and feeding it in at several points through cross-attention, so the denoising at every resolution level can consult the words. Classifier-free guidance means the model is trained on some examples with the condition dropped out, and at generation time the two predictions, with and without the condition, are extrapolated away from each other by a scale factor. That scale is the single most misused number in the field, and the section on pitfalls returns to it.

The remaining parts of a pipeline are separable components. The scheduler decides the noise schedule and the update rule at each step. Different samplers and schedulers take different numbers of steps to reach the same output, which is why a generation is a function of the model, the scheduler, the step count, the guidance scale and the seed, not of the model alone.

Why latent diffusion changed what was possible

Running that loop on raw pixels is expensive: a 512 by 512 RGB image is a quarter of a million values, and the network processes several of them per step. Latent diffusion moves the loop into a compressed space. A small autoencoder encodes an image into a much smaller tensor, the diffusion model runs there, and the decoder reconstructs the image at the end. The compression factor is what makes a 512 by 512 generation fit on a consumer GPU, and it is why the family of models carrying the Stable Diffusion name can be run locally at all.

Two capabilities follow from being able to start from something other than pure noise, which is where much of the tooling on this page comes from. Image-to-image starts from a noised version of an existing image, so the result is that image transformed rather than a new composition. Inpainting and outpainting do the same with a mask, adding noise only in the region to be regenerated, and ControlNet is the general form of that idea: our analysis of lllyasviel/ControlNet describes the original implementation of the architecture from the paper on adding conditional control to text-to-image diffusion models, adding precise spatial control to Stable Diffusion 1.5 through a dual-encoder design where one copy of the encoder is frozen and one is trained on conditioning data such as edge maps, depth maps, poses and segmentation masks.

The same separability explains why the ecosystem is split the way it is. A checkpoint, a text encoder and a decoder are separate downloads that must match, the scheduler is chosen at runtime, and control mechanisms attach to a frozen copy rather than replacing it. Assembling a pipeline is therefore a real task with a real failure mode, which is what the user interfaces below exist to hide.

When diffusion fits, and when it does not

Diffusion is the right tool when you have no paired data and want a distribution rather than a specific answer.

Where it fits. Image, video and audio generation where the only supervision is the raw media itself. Work that needs variation from the same prompt, since the sampling process produces a different sample each time from the same seed and settings. Editing tasks that are local rather than global, since a mask and a prompt express that requirement directly. Domains where the training data is collected, not labelled, which is the practical reason diffusion replaced supervised image generation in most settings.

Where it does not. Exact reproduction. A seed reproduces a result only with the same model, scheduler, step count, guidance scale and often the same software version, which makes it a reproducibility mechanism rather than a fidelity guarantee. Latency. Generation is a loop over many steps, and a real-time or interactive budget rules most of it out. Anything with a compact, deterministic answer, where a regression model, a search or a deterministic edit is both faster and better. And deployments where the model is too large to host, where an autoregressive or distilled alternative is the honest answer.

One non-technical limit is worth stating because it is a legal one rather than an engineering one. A checkpoint's licence is separate from the code that runs it, and the training data behind a released checkpoint is a question about provenance rather than about architecture.

The mistakes people actually make

Five of them account for most bad output.

The guidance scale. Classifier-free guidance extrapolates between the conditional and unconditional predictions, and pushing the scale too far produces the characteristic look of blown highlights, oversaturated colour and melted detail. Too low produces flat, washed-out images that ignore the prompt. The number is not a quality setting; it is a distance from the model's own distribution, and different schedulers respond to it differently.

Changing everything at once. Comparing a new scheduler, a new step count and a new scale in one experiment tells you nothing about which change helped. The seed is the only thing holding the rest constant, and it only holds it while nothing else moves.

Mismatched components. A checkpoint written for one latent space, a text encoder of the wrong length, or a scheduler from a different family produces a plausible-looking image that is subtly wrong, or an error at load time. This is the single most common support problem in this ecosystem and it is entirely a packaging issue.

The step count trade. More steps raise quality up to a point and then cost time for nothing. The right number depends on the scheduler, and the literature's defaults are not tuned for your card.

Confusing a model with a pipeline. What a user calls Stable Diffusion is a specific set of weights plus an autoencoder plus a text encoder plus a scheduler plus a sampler, and a reproduction attempt that gets one of them wrong will land near the image without being it.

How the projects divide

The repositories here fall into five groups, and the group tells you what question each one answers.

The reference implementations. CompVis/stable-diffusion holds the original v1 checkpoints, a reference txt2img script and a safety checker, and our analysis of it is that it is a research artifact rather than a product, with the README explicit about that. lucidrains/denoising-diffusion-pytorch goes further back: our analysis describes it as a DDPM in about the smallest API that still trains, a Unet, a GaussianDiffusion wrapper and a Trainer that reads a folder of images, and it is explicit that this is research and teaching code rather than a production image generator.

The library. huggingface/diffusers is the Apache-2.0 Python library for pretrained diffusion pipelines with interchangeable schedulers and reusable model blocks. Our analysis describes it as built for people who want to load a checkpoint in a few lines and then take the pipeline apart when they need to, which is the right characterisation: it is the layer where components are swappable rather than fixed.

The interfaces. AUTOMATIC1111/stable-diffusion-webui turns a Python and PyTorch install into a browser UI with txt2img, img2img, inpainting and an extension system, and our analysis tells you to treat it as a stable, slow-moving codebase rather than a fast one. Comfy-Org/ComfyUI is a node graph where the pipeline is a document you build, and our analysis puts the trade plainly: control over every model and parameter against assembling the pipeline yourself. invoke-ai/InvokeAI is a locally hosted application with a canvas and a gallery alongside node-based pipelines, on an Apache-2.0 licence with a separate launcher installer. Acly/krita-ai-diffusion is the opposite end, a GPL-3.0 Krita plugin that puts Stable Diffusion, Flux and ControlNet behind a panel inside a painting program, and our analysis says it is built for painters who want to keep their canvas rather than for people who want to tune sampler settings.

Training and deployment. ostris/ai-toolkit is a fine-tuning suite targeting LoRA adapters for FLUX, SDXL, Wan video and audio models on consumer-grade GPUs, shipping a GUI, a CLI, a Docker image and platform-specific launch scripts. apple-aiml-research/ml-stable-diffusion is the deployment side, converting PyTorch checkpoints into Core ML packages and shipping a Swift pipeline for iOS and macOS, and our analysis calls it a conversion and deployment toolchain rather than a desktop generator, with the hardware and OS floors stated in its README.

In practice

A diffusion model learns to reverse a fixed noising process, and everything a practitioner touches afterwards, the latent space, the scheduler, the guidance scale, the seed, is a parameter of the sampling loop rather than of the model. Start with huggingface/diffusers to understand what the pipeline is made of, use lucidrains/denoising-diffusion-pytorch or CompVis/stable-diffusion to see the mechanism with nothing in the way, and pick an interface only once you know which components you are changing. If you need spatial control, ControlNet is the piece that was built for it; if you need it on a phone, the Core ML conversion path is a separate problem from running it on a workstation; and if you need to train, ai-toolkit is aimed at adapters on consumer hardware rather than full training runs.