# DyRef attacks reference count: two stages, a reward reshape, and no saturation at 14K > The ECCV 2026 oral release that trains image editing models on many-reference prompts, where open-source models degrade as references multiply and diversify. Training is split into SFT and RL, with a LoRA format conversion standing between them. **Weistrass/DyRef** — [ECCV 2026 Oral] Official repository for Scaling Multi-Reference Image Generation with Dynamic Reward Optimization - Repository: https://github.com/Weistrass/DyRef - Stars: 354 · Forks: 9 - Language: Python - License: Apache-2.0 - Published: 2026-09-18 · Updated: 2026-09-18 · Language: en - Canonical page: https://hysenlabs.com/projects/weistrass-dyref ## The failure is reference count, not image quality The problem this project names is specific: current open-source image generation models degrade substantially as the number of reference images goes up and as the diversity of reference types expands. Nothing here is about resolution or artistic quality. That degradation is what blocks professional use, and two applications are named as the casualties, namely creative image generation and video keyframe generation. Both need a subject to stay consistent across several supplied references at once, which is exactly the case where a model trained mostly on single-image editing starts to fall apart. The approach is a training recipe rather than a new architecture. DyRef is described as a two-stage training framework, and the released weights are LoRA adapters for an existing base model rather than a checkpoint of their own. The repository carries separate directories for SFT, RL, model inference, benchmarks, data and result generation, which is the shape of a pipeline rather than a library. ## OmniRef-Bench scores combinations of types and counts The project introduces a benchmark of its own rather than relying on existing ones. OmniRef-Bench evaluates complex multi-reference image generation across combinations of reference-image types and quantities, which means a two-image case and a six-image case with mixed kinds of reference are scored on the same scale. The stated finding from running it is blunt: current open-source models still struggle substantially on it. Later claims are measured against a wider set. The multi-reference results are reported on OmniRef-Bench, MultiBanana and OmniContext, and single-image editing results on ImgEdit and DreamBench++, with the claim that the gains on the multi-reference side do not compromise the base model's other capabilities. Trained this way, Qwen-Image-Edit-2511 is reported as comparable to the closed-source Nano Banana Pro and ahead of Seedream 4.5, with FLUX.2-klein-base also improved. One presentation detail matters when reading the figures: the prompts shown in the results gallery are abbreviated for visualisation and are not the prompts actually used at inference. ## Stage I is supervised fine-tuning, Stage II changes the reward The two stages have clearly different jobs. Stage I is SFT, used to give the model the basic capability of handling complex multi-reference tasks at all. Stage II is where the method lives, and it has two parts. DRS enlarges the reward differences across samples, which spreads the training signal so that samples are not all treated alike. DAR shifts the model's focus onto the samples that carry numerous mixed-type reference images, which is the case the benchmark cares about most. Read together, that is a reward-shaping argument rather than an architecture argument: the model is pointed at the hard region of its own training distribution instead of being scaled up and hoped for. One caveat sits next to the data scaling result. Training sets of roughly 5K, 7.5K and 14K samples were evaluated and performance rose with size without clear saturation, but the text attributes the absence of larger runs to limited computational resources rather than to a decision. ## Three conda environments, because the dependencies disagree Installation is deliberately not one environment. SFT, RL and benchmark evaluation each get their own, because each component carries its own dependency requirements and merging them is not attempted. The SFT environment installs the package in editable mode from the SFT directory: ```bash conda create -n dyref_sft python=3.11 -y conda activate dyref_sft cd DyRef/sft pip install -e . ``` The RL environment is the same shape with the deepspeed extra: ```bash conda create -n dyref_rl python=3.11 -y conda activate dyref_rl cd DyRef/rl pip install -e .[deepspeed] ``` Python 3.11 is what both environments are created with. Inference runs in the SFT environment, so the RL environment is only needed for people retraining rather than only for people using the released weights. ## A conversion script stands between the two stages Running the pipeline is not two commands. The LoRA weights that SFT produces cannot be loaded by the RL code as they are, because the upstream repositories used for SFT and RL store LoRA weights in different formats. A conversion step is therefore required before the second stage can read the first stage's checkpoint. ```bash cd DyRef/sft python all_scripts/diffusers_peft_transfer.py --mode d2p \ --input /path/to/sft_checkpoint.safetensors \ --output /path/to/output_peft_format_dir \ --prefix transformer \ --verify ``` The mode name marks the direction, from the diffusers layout to the PEFT layout, and the verify flag is there to check the result rather than trust it. The three stages then run in order. Stage one is a shell script in the SFT directory: ```bash conda activate dyref_sft cd DyRef/sft bash all_scripts/Qwen-Image-Edit-2511_lora.sh ``` Stage two switches to the RL environment and invokes a script from DyRef/rl. The filename shown here is cut off partway through, and it packs the configuration into the name, carrying the rank, an added sample count, a CSD term and a SigLIP v2 reward variant. What follows the name in this file is not visible. ## Three models are always needed and one is conditional Weights come from Hugging Face rather than the repository. The list names Qwen-Image-Edit-2511, siglip2-base-patch16-384 from Google, dinov2-base from Facebook and clip-vit-large-patch14 from OpenAI, with FLUX.2-klein-base-9B from Black Forest Labs marked as something to fetch only if you need it. The first of those is the base image editor the method trains, and the middle two are what a reward or reward-adjacent model is built from. The download itself is one command with a local directory: ```bash hf download Qwen/Qwen-Image-Edit-2511 --local-dir "your path to save this model" ``` Inference then needs two things rather than one: the base model and the pretrained LoRA weights the project publishes separately under Weistrass/Qwen-Image-Edit-2511-DyRef. ```bash conda activate dyref_sft python model_inference/Qwen-Image-Edit-2511.py ``` Inference sits in the SFT environment, which is consistent with there being no third environment needed just to run a released adapter. ## Every planned item on the todo list is checked off The release is unusually complete for a paper repository, and the checklist is the clearest evidence of that. All seven boxes are marked done: the paper on arXiv, the training code for both Qwen-Image-Edit-2511 and Flux2-klein-base, the training data used in the work, OmniRef-Bench, the trained model weights, a pass at code efficiency and structure, and the project page. The news entries set the sequence. The paper was accepted on 2026.06.18, the training and inference code shipped on 2026.07.02 together with the data, benchmark and weights, and the paper was selected as an oral presentation on 2026.07.24. The authors are from Harbin Institute of Technology in Shenzhen with two independent researchers and Shenzhen Loop Area Institute. The last commit on the main branch is dated 2026-08-05, and no tagged release has been published, so the code state and the paper state are aligned by date rather than by version. ## Conclusion DyRef fits a team that already runs Qwen-Image-Edit-2511 or FLUX.2-klein-base and needs more than one or two reference images to hold together in one generation. It does not fit anyone expecting a pip install, since the project spans three conda environments and a conversion step between its stages. Before scaling up, note that the dataset study stops at 14K samples and the larger case is explicitly untested, so treat the absence of saturation as encouraging rather than established. Inference needs both the base model and the released LoRA weights, and the weights are the part that carries the result. ## FAQ ### What does DyRef actually change in the training pipeline? DyRef is a two-stage framework. Stage I is SFT for basic multi-reference capability, and Stage II uses DRS to enlarge reward differences across samples and DAR to focus on samples carrying many mixed-type reference images. ### Why does DyRef need a weight conversion step between its two stages? The upstream repositories used for SFT and for RL store LoRA weights in different formats, so a conversion is required. The script diffusers_peft_transfer.py runs with mode d2p and a verify flag to check the output. ### Which models does the DyRef setup fetch from Hugging Face? Qwen/Qwen-Image-Edit-2511, google/siglip2-base-patch16-384, facebook/dinov2-base and openai/clip-vit-large-patch14, plus black-forest-labs/FLUX.2-klein-base-9B if you need it. ### What is OmniRef-Bench? It is the benchmark introduced with DyRef to evaluate complex multi-reference image generation across combinations of reference-image types and quantities. The reported finding is that current open-source models still struggle substantially on it. ### How large was the training set used for DyRef? Approximately 5K, 7.5K and 14K samples were evaluated, with performance improving as size grew and no clear saturation. Larger sets were not run, which the project attributes to limited computational resources. ### What did the DyRef release include? Training and inference code released on 2026.07.02 along with the training data, the benchmark and model weights. The pretrained LoRA weights are published separately as Weistrass/Qwen-Image-Edit-2511-DyRef. ## Sources - [Issues](https://github.com/Weistrass/DyRef/issues) - [License: Apache-2.0](https://github.com/Weistrass/DyRef/blob/main/LICENSE) - [README](https://github.com/Weistrass/DyRef/blob/main/README.md) - [Weistrass/DyRef on GitHub](https://github.com/Weistrass/DyRef) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/weistrass-dyref