HY-SOAR corrects diffusion trajectory errors without a reward model

The problem HY-SOAR names: exposure bias in the denoising trajectory

HY-SOAR, for Self-Correction for Optimal Alignment and Refinement, is a reward-free post-training method for rectified-flow diffusion models, and the README states its aim in one line: beyond SFT and RL, self-correction during generation without reward models, preference labels or negative samples.

The technical target is exposure bias in the denoising trajectory. Standard supervised fine-tuning trains the denoiser on ideal forward-noising states derived from real data, while inference conditions on states produced by the model's own earlier predictions. Once an early denoising step drifts, later steps must recover from states that were never directly optimised, and the README says errors can compound across the trajectory.

That is a specific and well known weakness of SFT for diffusion models, and it is worth being precise about the claim being made. HY-SOAR does not argue that SFT is wrong. It argues that SFT optimises the wrong states, the ones the model would have seen if it had been perfect, and that the states the model actually visits at inference time are different.

The alternative framing in the introduction is the timing of the feedback. Instead of waiting for a terminal reward after a full rollout, SOAR teaches the model to correct its own trajectory errors at the timestep where they occur.

That reframing is the whole idea, and it explains the three key features the README lists. The method claims an on-policy, dense and reward-free training signal, and it does so by generating the training states from the model being trained rather than from a fixed dataset of ideal states.

The context matters for judging the work. This comes from Tencent Hunyuan, and the README links a report on arXiv at 2604.12617, a project page at hy-soar.github.io, and a Hugyuan account. The April 2026 news entry says the training and evaluation code is publicly available.

The three-step mechanism, step by step

The algorithm is compact enough to state precisely, and the README does.

Given a clean latent z_0, a noise endpoint z_1 and a condition c, SOAR does three things.

First, it samples an on-trajectory noisy state and performs one stop-gradient CFG rollout step with the current model. The stop-gradient matters: the step is used to produce a state, not to propagate gradients through the model's own rollout.

Second, it re-noises the resulting off-trajectory state toward the same noise endpoint z_1 to create auxiliary states. Re-noising toward the same endpoint rather than a fresh one is the geometric choice the README highlights, and it keeps the auxiliary states near the original transport ray, yielding a concrete correction velocity anchored to z_0.

Third, it supervises the denoiser with the analytical correction target, written in the README as v_corr = (z at the new sigma minus z_0) divided by that sigma.

The consequence of the second and third steps together is the on-policy property. Off-trajectory states are produced by the current model's own rollout, so the training distribution co-evolves with the model instead of staying fixed to the SFT data trajectory.

And the consequence of the third step is density. Instead of one terminal signal for a whole sample, the objective produces per-timestep correction supervision, which is what the README means when it says the method avoids terminal-reward credit assignment.

The base objective also subsumes standard SFT, and the auxiliary correction loss is what trains on nearby model-induced states. That is why the README describes SOAR as a stronger first post-training stage that remains compatible with subsequent reward-based alignment, and why the fourth key feature is called compatible post-training stage.

The last point is the strategically interesting one. If the method works, it is meant to sit in front of the RL stage rather than replace it, which means the evaluation of interest is not only whether SOAR beats SFT but whether a pipeline of SOAR then reward optimisation ends up better than reward optimisation alone.

Setting up the environment, and the base model you must accept

The implementation is based on the DiffusionNFT and Flow-GRPO codebases, with most environments aligned, and the install sequence is given in full.

git clone https://github.com/Tencent-Hunyuan/HY-SOAR.git
cd HY-SOAR

conda create -n hy-soar python=3.10.16
conda activate hy-soar
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu126
pip install -e .
export PYTHONPATH=$PWD/sora:$PYTHONPATH

Two details in that block deserve attention. The PyTorch install pins the CUDA 12.6 wheel index explicitly, so the environment assumes an NVIDIA GPU and a specific CUDA generation. And the PYTHONPATH export points at a directory named sora inside the repository, which is where the training code lives, matching the top level entry of a soar directory with an assets directory, a setup.py and the LICENSE.

The base model is specified next. The training script expects stabilityai/stable-diffusion-3.5-medium as the pretrained model, and you need to accept the model license on Hugging Face and authenticate via huggingface-cli login before training.

That is a real gate rather than a formality. SD 3.5 Medium is a gated model, so the licence acceptance and the token have to be in place before a training run can even load the weights.

The dependency set in setup.py is pinned hard, and the pins tell you what the code expects:

    "torch==2.6.0",
    "torchvision==0.21.0",
    "transformers==4.40.0",

The rest of the list is long and specific: diffusers 0.33.1, accelerate 1.4.0, numpy 1.26.4, wandb 0.18.7, flash-attn 2.7.4.post1, deepspeed 0.16.4, peft 0.10.0, bitsandbytes 0.45.3, fastapi 0.115.11 with uvicorn, and huggingface-hub 0.29.1. The package name is hy-soar at version 0.0.1, python_requires is greater than or equal to 3.10, and a dev extra pulls in ipython, black and pytest.

Two things in that list deserve a second look for anyone planning a cluster. flash-attn and deepspeed mean this is built for multi-GPU training, and the presence of fastapi and uvicorn means the reward models are served locally over HTTP rather than called out to a remote endpoint.

Reward models are supported but not required

The reward preparation section lists the reward models the implementation supports: GenEval, OCR through PaddleOCR, PickScore, ClipScore, HPSv2.1, Aesthetic and ImageReward.

That list is interesting precisely because the method claims not to need a reward model. The role of these models in this repository is evaluation and, presumably, the subsequent reward-based alignment stage that HY-SOAR is designed to precede, not the SOAR loss itself.

Two implementation notes are worth keeping. The README says it additionally supports HPSv2.1 on top of FlowGRPO, and that it simplifies GenEval from a remote server to a local setup. Running GenEval locally matters in practice, because a remote evaluation server is a network dependency and a rate limit in the middle of a training or evaluation loop.

The checkpoints are downloaded rather than installed. The README gives the commands:

mkdir reward_ckpts
cd reward_ckpts
# Aesthetic
wget https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/refs/heads/main/sac+logos+ava1-l14-linearMSE.pth

and then the equivalent for GenEval, ClipScore and HPSv2.1, with weights coming from the improved-aesthetic-predictor repository, an OpenMMLab download host for the GenEval mask2former checkpoint, and Hugging Face for the CLIP and HPSv2 files.

So the evaluation side of this project is a substantial download, several gigabytes across seven reward models, and a set of Python packages that have to interoperate with each other. That is the price of the reward-based comparison the README presents.

The comparisons themselves are described as pictures rather than tables. The aesthetic comparison is framed as SOAR against Flow-GRPO and SFT across training steps on historical scenes, fantasy art and character portraits. The CLIPScore comparison is on design and poster prompts for text-image alignment, where the README claims stronger text rendering and compositional fidelity. The third comparison covers web user interface and graphic design generation, with claims about layout, typography and visual hierarchy.

All three are image comparisons, and none of them comes with a numeric table in the README.

What is in the repository, and what the table of contents promises

The top level of the repository is small: .gitignore, LICENSE, README.md, assets, setup.py and the soar directory. The assets directory holds the teaser and comparison images the README embeds, and the soar directory is the code the PYTHONPATH export points at.

The README's table of contents is longer than the content in the file we have, and the difference is worth noting rather than glossing over. The contents list sections for environment setup, reward preparation, usage, evaluation, data format, citation and acknowledgement, in addition to the introduction, key features and the comparisons. The sections on usage, evaluation and data format are listed but their body content is not included in the README text that was provided, so the exact training invocation, the evaluation entry point and the expected data layout are not described there.

That gap matters for anyone planning to reproduce the work, because the reward preparation and the model download are fully specified while the actual command that starts training is not. You can install the environment and fetch the checkpoints from the README alone, and then you need the repository itself, or the paper, for the rest.

The open-source plan section is short and indicates the scope of the release. HY-SOAR has two checkboxes ticked, training code and evaluation code. There is no checklist for data, no pretrained checkpoints of a SOAR-tuned model, and no serving or inference tooling in the release.

The acknowledgement section and the citation entry point to the arXiv report, and the project page badge links to hy-soar.github.io, which is where the material not in the repository is expected to live.

Where HY-SOAR is the wrong tool

The limits here are mostly about maturity and reproducibility rather than about the idea, and it is fair to separate the two.

The first limit is the evidence. The comparisons in the README are images of results, not numeric tables, and there is no benchmark section with scores, no ablation table and no reported reproduction. If your decision process needs a number to compare against another method, this repository does not provide one. The report on arXiv is where to look for that.

The second limit is the base model. The training script expects stabilityai/stable-diffusion-3.5-medium, and the method is defined for rectified-flow diffusion models. If your target is an autoregressive image model, or a different architecture, or a much larger flow model that the environment is not set up for, the code as shipped is not the path.

The third limit is the environment. Every dependency is pinned to an exact version, including torch 2.6.0, flash-attn 2.7.4.post1 and deepspeed 0.16.4. Pinned stacks are reproducible and they are also brittle: a driver or CUDA change that is fine for a normal project breaks this one. The setup is also NVIDIA specific, with the CUDA 12.6 wheel index and a flash-attn build in the dependency list.

The fourth limit is the scope of the release. This is training and evaluation code for a research method. There is no inference service, no packaged model weights and no supported production path. Anyone wanting to run HY-SOAR in a product is building the deployment themselves on top of a research repository.

And the licensing needs reading. The licence is shown in the repository metadata as Other, a custom licence GitHub cannot classify, so the LICENSE file at the root is the text that governs. The repository is derived from DiffusionNFT and Flow-GRPO, and the README acknowledges those codebases, so the terms of the upstream projects are part of the picture. The package version is 0.0.1, the repository publishes no GitHub releases, and the last push was on 2026-04-21, so there is no release history to track against.

Editorial conclusion

Research teams post-training a rectified-flow diffusion model on a fixed compute budget should read HY-SOAR, because it claims a denser signal than SFT with no reward model, no preference labels and no negative samples, and the training and evaluation code is public rather than a paper-only result. It is the wrong tool if you need to reproduce the exact numbers, since the README gives qualitative comparisons rather than a benchmark table, and it is the wrong tool if your base model is not a rectified-flow model, since the method is defined in those terms. Before running it, clone the repository and install with the pinned torch 2.6.0 and diffusers 0.33.1 environment, accept the stable-diffusion-3.5-medium licence on Hugging Face and run huggingface-cli login, and note that the package version is 0.0.1 with no tagged releases.

Frequently asked questions

Official sources

1. Issues
2. README
3. Tencent-Hunyuan/HY-SOAR on GitHub