SRPO: aligning a diffusion model's whole trajectory with a reward, without rollouts

What the method changes

The paper is titled Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference, it is on arXiv as 2509.06942, and the weights are published on Hugging Face as tencent/SRPO with a project page alongside.

The first key feature, Direct Align, is described as a new sampling strategy for diffusion fine-tuning that can effectively restore highly noisy images, leading to an optimization process that is more stable and less computationally demanding, especially during the initial timesteps.

That phrase about highly noisy images is the mechanism. In a diffusion model, the early steps of sampling are the noisy ones, and a method that can undo the damage there has a much shorter path back to a good image than one that has to repair it at the end. The consequence the authors claim is a more stable optimisation that costs less.

The authors are from Hunyuan at Tencent, with affiliations at the School of Science and Engineering of the Chinese University of Hong Kong, Shenzhen and the Shenzhen International Graduate School at Tsinghua University, and the paper lists an equal contribution marker and a project lead.

The repository is not archived and the last push was on 2026-05-11, with no GitHub releases, so the published artefacts are the Hugging Face checkpoint, the ComfyUI workflow and the training code in the repository.

Why the rollouts go away

The second key feature is Faster Training, and its mechanism is the one that distinguishes the method from the reinforcement learning it replaces.

By rolling out only a single image and optimising directly with analytical gradients, a key distinction from GRPO, the method achieves significant performance improvements for FLUX.1.dev in under ten minutes of training.

In online reinforcement learning for diffusion models, each optimisation step requires generating a batch of images with the current policy and scoring them with a reward model. That generation is the expensive part, and it is also the part that cannot be differentiated through, which is why the algorithms in this family fall back on score estimates rather than gradients. Removing the rollout loop removes the cost, and doing it with one image instead of a batch is what makes the direct backpropagation possible.

The second half of the feature is more striking still. The method supports replacing online rollouts entirely with a small dataset of real images, and the authors find that fewer than 1500 images are sufficient to effectively train FLUX.1.dev.

Fifteen hundred images is a fine-tuning dataset size rather than a training corpus, and that is the practical headline. The method moves the bottleneck from generation to data, and for most teams that is a better trade.

Reward hacking, and how they claim to avoid it

The third key feature is Free of Reward Hacking, and this is the one that needs the most careful reading.

The claim is two-part. The training strategy is improved for methods that use direct backpropagation on the reward signal, such as ReFL and DRaFT. And the model is directly regularised using negative rewards, without the need for a KL divergence term or a separate reward system.

The negative reward regularisation is the interesting idea. In a method that maximises a reward by backpropagating through the reward model, the easiest way to raise the reward is often to find a shortcut in the reward model itself. A typical failure is overfitting to one visual attribute, and the authors name colour and oversaturation preferences as the examples of the kind of artefact this produces. By pushing the model away from low-reward images as well as towards high-reward ones, the optimisation is less able to satisfy the reward by moving in a direction the reward model happens to like.

The claim is that in their experiments the approach achieves comparable performance with multiple different rewards, while improving the perceptual quality of FLUX.1.dev without the reward hacking problems. Comparable performance across different reward models is the evidence that matters here, because a method that only works with one reward is exactly the method that has learned that reward's quirks.

The fourth feature is a potential rather than a shipped capability. For the first time in online RL, they incorporate dynamically controllable text conditions, enabling on-the-fly adjustment of the reward preference towards styles within the scope of the reward model.

The open source plan, and what is still missing

The plan is a three-line checklist, and it is worth reading for what is not ticked.

Done is the training code for FLUX.1-dev, and done is support for Qwen-Image. Not done is extending support to other models, with Qwen-Image-Edit and FLUX 2 named as the examples.

So the honest scope of the released code is two base models, and the news section confirms the order things arrived in. On 2025-09-08 they released the paper, the checkpoint and the inference code. On 2025-09-12 they released the complete training code, plus tips and experiences to help you train your models, and a standard workflow that is said to work in ComfyUI. On 2025-10-26 they reported reaching the top position on the Artificial Analysis leaderboard for text-to-image open source models. And on 2026-02-24 they released the training code for online and offline training built upon Qwen-Image.

The distinction in that last entry is worth keeping. Online training is the reinforcement learning loop with a reward model, and offline training is the variant that uses the small dataset of real images. Both are released.

The repository also ships a `prompts.txt` and a `videos2caption2.json`, which are the evaluation and captioning side of the experiments rather than the training code, and an `assets/` directory with the figures used in the README.

Setting up the environment, and the dependency pinning

The installation section is short, and the environment is created with conda:

conda create -n SRPO python=3.10.16 -y
conda activate SRPO
bash ./env_setup.sh

The README notes that the environment dependencies are essentially the same as DanceGRPO, which is a useful pointer for anyone who has already set that project up.

The Python version is pinned exactly, 3.10.16, which is worth noticing in a project whose dependency list is as old as this one.

The `pyproject.toml` explains why the setup is heavier than it looks. The project is named fastvideo at version 1.2.0, and the repository has a `fastvideo/` directory, so the training code is built on the FastVideo project rather than being a standalone implementation. The dependency list is long and pinned to exact versions, with transformers at 4.46.1, diffusers at 0.32.0, accelerate at 1.0.1, pytorch-lightning at 2.4.0, peft at 0.13.2, liger_kernel at 0.4.1, wandb, moviepy, decord and pytorchvideo among them. There is also a `requirements-lint.txt` for the lint environment, which is a small sign of a codebase that is checked rather than merely run.

Inference, and the ComfyUI workflow

There are two ways to run the model, and the second is the one most people will start with.

The first is ComfyUI. The README says you can use SRPO in ComfyUI, and gives a workflow JSON file, `comfyui/SRPO-workflow.json`, that you can load directly, with a tip that the workflow JSON information is embedded in the image file, so you can load the picture and get the graph.

The second is a plain diffusers pipeline, and the code is short enough to read as the specification of what the checkpoint expects:

import torch
from diffusers import FluxPipeline
from safetensors.torch import load_file

pipe = FluxPipeline.from_pretrained('./data/flux',
        torch_dtype=torch.bfloat16,
        use_safetensors=True
    ).to("cuda")
state_dict = load_file("./srpo/diffusion_pytorch_model.safetensors")
pipe.transformer.load_state_dict(state_dict)

The important detail is that the SRPO checkpoint is a state dict loaded into the transformer of a base pipeline, not a pipeline of its own. The base model is `black-forest-labs/FLUX.1-dev`, and the weights are fetched with the huggingface-cli into `./data/flux`, while the SRPO checkpoint goes into `./srpo`. The inference parameters are a guidance scale of 3.5, 1024 by 1024, 50 steps and a max sequence length of 512.

For batch work, the README gives a torchrun invocation over `vis.py` with the node rank and rendezvous settings, and says to replace `model_path` in that script before running it.

Where SRPO is the wrong choice

The first wrong case is a team that wants to fine-tune a model outside the two supported bases. Extending support to other models, with Qwen-Image-Edit and FLUX 2 named, is still unchecked on the open source plan, and the code is built on FastVideo and a pinned diffusers, so a new base model means real work rather than a configuration change.

The second is a team that wants the full online reinforcement learning setup. The method removes the rollout loop, which is where the reward model, the policy optimisation and the controllable text conditions would live. The fourth key feature, dynamically controllable text conditions that let you steer the reward preference towards a style at run time, is described as a potential, so a project that plans to use it is planning around something that is not in the released code.

The third is a team that cannot reproduce the training. The dependencies are pinned to versions from 2024, the Python version is pinned to 3.10.16, and the environment is created with a conda command and an `env_setup.sh` script. On a modern machine with a current diffusers and torch stack, this will need work.

The fourth is a project with a licensing question about a model derived from a base with its own terms. The repository's licence field is recorded as a custom one, and the `pyproject.toml` classifier says Apache Software License, so the licence file itself is what to read, and separately FLUX.1-dev is a gated model with its own licence that your organisation will also have to accept.

The fifth is anyone who thinks the leaderboard position transfers to their own prompts. The reported result is a top position on one leaderboard for text-to-image open source models, measured with one reward model family, and the paper's own claim is comparable performance across different rewards, not universal superiority.

Editorial conclusion

SRPO suits a team that wants to fine-tune a text-to-image diffusion model against a human preference reward and has been put off by the cost of reinforcement learning rollouts, because the method removes the rollout loop and the authors report training in under ten minutes. It does not suit a team that needs a model outside the two supported bases, since extension to Qwen-Image-Edit and FLUX 2 is still unchecked on the open source plan, and it does not suit a project that needs the full online RL machinery, because the controllable text condition is described as a potential rather than a shipped feature. Before adopting it, read the `env_setup.sh` script and the pinned `pyproject.toml` rather than assuming a current diffusers stack will work, and check the `fastvideo/` directory, since the training code is built on that project rather than being standalone.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Tencent-Hunyuan/SRPO on GitHub