DiffusionOPSD detaches the reward graph and refreshes the behaviour policy with an EMA

The gap is at intermediate predictions, not the final image

The problem statement is specific and it is what distinguishes this from ordinary reward tuning. Diffusion reward optimisation observes an image-level score only after a multi-step denoising trajectory has finished, which leaves a supervision gap at every intermediate prediction: the model is told whether the finished image was good, but never what a better image would have looked like at step three of thirty. The approach here is to close that gap with an on-policy self-distillation loop. Two backbones are covered, SD3.5-M at 512 squared and Z-Image-Turbo at 1024 squared in its native few-step regime, which is worth noting because a method that only works at low resolution and low step counts would not be much use for the second backbone. The reported result is the best final held-out score in 19 of 20 reward-matched settings, with training GPU-hours reduced against DiffusionNFT by 40 percent and 63 percent on the two backbones respectively.

Anchors from the frozen policy, targets from reward gradients

The loop has three moving parts and the separation between them is the design. A frozen behaviour policy runs a trajectory, a low-noise state along it is selected as the query, and the policy's own clean-output prediction at that state becomes the anchor, detached so nothing flows back through it. From that anchor, two sets of targets are constructed by taking normalised reward-gradient steps in opposite directions inside a relative trust region: one step up the reward and one step down, each normalised by its own gradient norm with an epsilon for stability. The trainable policy is then asked to fit those targets through a pair of branches, positive and negative, with the loss a weighted sum of normalised squared residuals. In the implementation those normalisers are detached mean-absolute residual scales and the residuals are elementwise means rather than full sums, and the whole objective is scaled by an advantage coefficient fixed at five. The minimal pseudocode shows the whole sequence, ending with the EMA update:

trajectory = rollout(behavior_policy, prompt)
query = select_low_noise_state(trajectory)
anchor = behavior_policy.clean_output(query).detach()
weight = group_normalized_endpoint_weight(trajectory.reward)
positive = bounded_reward_ascent(anchor, reward).detach()
negative = bounded_reward_descent(anchor, reward).detach()
prediction = trainable_policy.clean_output(query.detach())
loss = detached_target_loss(prediction, anchor, positive, negative, weight)
loss.backward()
optimizer.step()
update_behavior_policy_ema()

Detaching the reward graph is the load-bearing detail

The third highlight is the one an implementer should not skip. Before the trainable policy is fitted, the reward and decoder graphs are discarded, and the targets enter as detached values. That is what the write-up means by separating target construction from finite realisation, and it does two things at once. It means the quality of the target is a property you can inspect on its own, rather than something entangled with however many gradient steps the optimiser managed to take. And it means the fitting step is a supervised regression problem against a fixed target, which is why the update budget can be finite without the method collapsing. On-policy query collection is the other half of the same idea: supervision is built on states the current behaviour policy actually visits, rather than on offline data or forward-noised substitutes, which is what stops the targets from drifting away from the distribution the policy is in. Continually refreshed supervision follows from the same mechanism, since an EMA of the behaviour policy regenerates the trajectories, the anchors and the targets after each outer update.

The efficiency numbers exclude setup and use eight GPUs

The cost table is the one to read with its footnote rather than its headline. It is eight-GPU profiles, and initialisation, calibration and evaluation are excluded, so it measures steady-state training only. On that basis the SD3.5-M comparison against DiffusionNFT is 126.9 seconds per update against 212.4, and 28.2 GPU-hours per hundred updates against 47.2, a relative factor of 0.60. The Z-Image-Turbo comparison is larger: 674.0 seconds per update against 1826.2, and 149.8 GPU-hours per hundred updates against 405.8, a factor of 0.37. The memory numbers move the other way, and that is the honest counterweight. Peak VRAM is 50.0 GB against 47.8 GB on SD3.5-M, and 61.5 GB against 49.9 GB on Z-Image-Turbo, so the method buys its speed with more memory per process rather than with less. Anyone planning capacity from these figures should keep both columns in view.

Where it wins, and the one cell it does not

Both result tables are reward-specific, with each checkpoint scored on the objective it was trained for, and the headline claim is the 19 of 20 count rather than any single number. On SD3.5-M the DiffusionOPSD row leads on PickScore at 24.94, CLIP at 0.340, HPSv2.1 at 0.390, ImageReward at 1.76, HPSv3 at 13.34, DeQA at 4.94, AltCLIP at 0.450, and the two preference metrics at 0.214 and 0.465. The exception is the aesthetics column, where ReFL scores 12.09 against 12.08, which is the cell that takes the total to 19 rather than 20. On Z-Image-Turbo the picture is cleaner and the row leads on every listed column, including 25.15 PickScore and 0.551 on the pairwise metric, against ReFL at 24.54 and 0.481. Presets are published for all seven open-weight evaluators and for arbitrary positive weighted sums, with the paper's own example combining PickScore/26, CLIPScore and HPSv2.1.

transformers is pinned to one minor, and the comments say why

The dependency file is where the practical friction lives, and the comments explain each constraint rather than leaving you to discover it. `transformers` is held between 4.51 and 4.52 because HPSv3 checkpoints use the pre-4.57 Qwen2-VL state-dict layout, and 4.51.x is described as the release's validated common stack. `huggingface-hub` is capped below 1.0 with the upper bound kept explicit, because Transformers 4.51 requires the pre-1.0 Hub API and an unrelated later install could otherwise break model imports. The `rewards` extra pulls fairscale pinned exactly, open-clip-torch and timm, with a note that Transformers 4.51 imports its timm wrapper while resolving AutoModel and that ImageReward's old package metadata pins timm 0.6.13, so ImageReward itself has to be installed with `--no-deps`. The rest is conventional: diffusers, accelerate, peft, torch and torchvision with floors rather than pins, datasets, wandb for tracking, and a dev group of pytest and ruff with a line length of 120.

src/ is the package, and there is a diffusers patch

The repository layout explains how much of this is upstream. The setuptools configuration maps the package directory to `src` and ships two packages, `diffusionopsd` itself and `diffusionopsd.diffusers_patch`, which tells you the project carries a patch against the diffusers library rather than working entirely through its public API. That is a reasonable shape for a post-training method that needs access to internals, and it is also a maintenance cost: the patch has to track diffusers releases, and the dependency file only sets a floor of 0.36 on it. Alongside `src/` at the top level sit `opd/`, `config/`, `data/`, `scripts/` and `assets/`, with no documentation site beyond the project page and no GitHub releases at all. The package itself is at version 0.1.0, requires Python 3.10 or newer, and the last push to `main` is dated 2026-08-27, which is the only version marker available since there is nothing tagged.

Editorial conclusion

DiffusionOPSD suits a team already doing reward-guided post-training of a diffusion backbone and stuck on the same wall, which is that the reward only speaks once at the end of a long denoising trajectory. The construction here gives you something to look at at every intermediate step, and the detachment is the part that makes the targets inspectable rather than entangled with the training graph. It is not a paper reproduction harness, since the note at the top says it is an external implementation of the published algorithm rather than the authors' code. Before you plan around the headline numbers, read the efficiency table's footnote: the GPU-hour figures come from eight-GPU profiles with initialisation, calibration and evaluation excluded, and the memory figures are higher than the baseline rather than lower. Then check the dependency pins, because `transformers` is held to a single minor version for a stated layout reason.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. worldbench/DiffusionOPSD on GitHub