# DiffusionDPO applies direct preference optimization to diffusion models

> DiffusionDPO is the training code for the paper on aligning diffusion models with direct preference optimization, built on diffusers with a single bulky train script and launcher shells. It is research code for reproducing a paper, not a library to build on.

**SalesforceAIResearch/DiffusionDPO** — Code for "Diffusion Model Alignment Using Direct Preference Optimization"

- Repository: https://github.com/SalesforceAIResearch/DiffusionDPO
- Website: https://arxiv.org/abs/2311.12908
- Stars: 715 · Forks: 52
- Language: Python
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/salesforceairesearch-diffusiondpo

## Research code for one paper, and it says so

The repository is described in one line as the training code for the Diffusion-DPO paper, and the ethical considerations section is unusually explicit about scope. It states that this release is for research purposes only in support of an academic paper, that the models, datasets and code are not specifically designed or evaluated for all downstream purposes, and that users should evaluate and address concerns related to accuracy, safety and fairness before deploying. The project also asks users to consider the common limitations of AI, comply with applicable laws, and follow best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly affect people. The repository carries a separate AI_ETHICS.md alongside a CODE_OF_CONDUCT.md, a CONTRIBUTING.md and a SECURITY.md, which is the file set you expect from a research release that expects outside contributions and outside bug reports. For an engineer, the useful reading is that this is code you adapt, not code you depend on. The licence is Apache-2.0 with the text in LICENSE.txt, which is permissive enough for commercial use, but the research-only framing and the reference to the AUP and AI AUP for use case guidance are conditions you should respect in spirit if you take the checkpoints.

## The training script is one file of over a thousand lines

The repository layout is small and the structure documentation is candid about it. train.py is the main script, described as pretty bulky at more than 1000 lines, with the training loop starting at around line 1000 at this commit, and the instructions tell you to search for "for epoch" to find it. That single detail tells you how to approach the code: this is a fork of a diffusers example script, not a designed API. The project description says the script is adapted from the diffusers library, specifically the text_to_image example from the v0.20.0 release. The rest of the tree is launchers/, which holds example shell scripts for running SD1.5 or SDXL training, utils/, which contains the scoring models used for evaluation or AI feedback, a quick_samples notebook, the requirements file, and upload_model_to_hub.py, described as a simple utility for uploading a model checkpoint to the Hugging Face Hub whose current values are placeholders. A monolithic training script adapted from an example is exactly the right shape for a paper reproduction, and exactly the wrong shape for a library. Anyone planning to build on it will end up forking the file.

## The launcher assumes a cluster, not a laptop

The example SD1.5 launch is the best single view of what the training run involves. It sets the model and dataset through environment variables, pointing at the runwayml/stable-diffusion-v1-5 checkpoint and the yuvalkirstain/pickapic_v2 dataset, and then launches train.py through accelerate in mixed precision fp16. Setup is a single pip install of the requirements file, and the launcher itself is short enough to read in full:

```bash
accelerate launch --mixed_precision="fp16"  train.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME \
  --train_batch_size=1 \
  --dataloader_num_workers=16 \
  --gradient_accumulation_steps=1 \
  --max_train_steps=2000 \
  --lr_scheduler="constant_with_warmup" --lr_warmup_steps=500 \
  --learning_rate=1e-8 --scale_lr \
  --cache_dir="/export/share/datasets/vision_language/pick_a_pic_v2/" \
  --checkpointing_steps 500 \
  --beta_dpo 5000 \
   --output_dir="tmp-sd15"
```

The comments carry the numbers that matter. The effective batch size is the product of the GPU count, the train batch size and the gradient accumulation steps, and the paper used 2048. The comment also says training takes about 24 hours over 2000 steps, and the checkpoint description explains that the published models are trained as described in the paper, replicable with the scripts in launchers/ assuming 16 GPUs, with gradient accumulation scaled accordingly. So the reference configuration is a 16 GPU job, and a single GPU run is not the intended path even though the flags allow it. The learning rate in the example is 1e-8 with --scale_lr, and the comment on that flag says it was found to be very helpful but is not the default in the code. Details like that are the reason to read the launcher rather than reconstruct the command from the argument list.

## Every knob in the DPO run is an explicit flag

The argument documentation is the most useful part of the repository, because it tells you what the script actually controls. On the general side there is the checkpoint you initialise from, the output directory, the training seed which is not set by default, a --sdxl switch to run SDXL training instead, and --sft to run supervised fine-tuning rather than DPO. The DPO specific arguments are --beta_dpo, described as the KL-divergence parameter beta for DPO, and --choice_model, the model used for AI feedback, with Aesthetics, CLIP, PickScore and HPS named as the options. The optimiser section covers max train steps, gradient accumulation, train batch size, checkpointing steps, learning rate, the scale_lr flag, the scheduler type and warmup steps, and --use_adafactor, which swaps Adam for Adafactor to lower memory use and is the default for SDXL. Gradient checkpointing is turned on automatically for SDXL. The data section names the dataset switch away from Pick-a-Pic, the local cache directory, the resolution, which defaults to 512 for non-SDXL and 1024 for SDXL, the random crop and horizontal flip augmentation switches, and the number of dataloader workers. Two of these deserve a second read. The cache directory note says users will want to change it to fit their file system, which is a hint that the shipped value points at one researcher's storage layout. And the resolution default differs between the two model families, so switching to SDXL changes the data pipeline as well as the model.

## Evaluation is built on reward models rather than human review

The evaluation side of the project is what makes it useful beyond a single training run. The utils/ directory holds the scoring models used for evaluation or for AI feedback, and they are named: PickScore, HPS, Aesthetics and CLIP. The choice model is selected with the --choice_model flag, so the same run can be driven by different scorers. The quick_samples notebook is described as visualizations from a pretrained model versus the baseline, and it also contains a sample of automatic quantitative evaluation using PickScore. That is the whole evaluation story in the repository, and its limits are worth stating plainly. Every metric here is a learned preference model, not a human study and not a safety evaluation. A PickScore or HPS number tells you that an image scores well against a human preference dataset, which is a useful signal for preference alignment and a poor signal for whether the model is safe, unbiased or factually sound. The project's own ethical section makes that distinction explicit by asking users to evaluate accuracy, safety and fairness separately before deploying. The dependency list confirms the stack, with hpsv2, openai_clip at 1.0.1, bitsandbytes, xformers at 0.0.22 and diffusers pinned at 0.20.0, alongside transformers 4.30.2 and accelerate 0.20.2.

## What a fine-tuning run actually costs you

The practical costs are hardware, time and dependency friction, and the repository is clear about all three. Hardware means 16 GPUs for the reference configuration, and the launchers are written on that assumption, with gradient accumulation scaled to compensate otherwise. Time means about 24 hours for 2000 steps at the paper's effective batch size of 2048, and the choice of Adafactor over Adam for SDXL is an explicit memory trade rather than an optimisation. Dependency friction means the requirements file is pinned to versions from the diffusers 0.20 era, with accelerate 0.20.2, diffusers 0.20.0, transformers 4.30.2, xformers 0.0.22 and openai_clip 1.0.1, plus torchvision, datasets, ftfy, tensorboard, Jinja2 and bitsandbytes. Those pins are what makes a paper reproducible and are also what will age worst, since a newer CUDA or Python will need you to move the pins or rebuild xformers and bitsandbytes from source. The scripts and the code assume 16 GPUs and a specific cache path, so the first honest task for anyone reusing this is to rewrite the launcher for their own machine and their own storage layout.

## Where DPO on images stops and something else starts

It is worth being clear about the boundary of this approach, because a preference-optimised image model is not the same thing as a model you can ship. Direct preference optimization here trains the model against a learned preference signal, and the project ships checkpoints for Stable Diffusion 1.5 and Stable Diffusion XL 1.0 that were initialised from those base models and trained as the paper describes. The alternative path for a team that wants preference-aligned image output is to start from a model that already had alignment applied by its authors, rather than running this training themselves, which trades away the ability to shape the preference data yourself but removes the 16 GPU job. A second alternative is reinforcement learning from human feedback on the image task with a proper human annotation pipeline, which is the method this project is a cheaper approximation of, and which is why the automatic evaluation with PickScore is a proxy and not a substitute. A third is not optimising the model at all and using prompt level techniques for aesthetic control. The choice comes down to whether the preference signal has to be yours. If it does, this repository is the reference implementation, and you should budget the cluster time accordingly.

## Conclusion

Adopt DiffusionDPO if you are reproducing the paper's results, fine-tuning Stable Diffusion 1.5 or SDXL on a preference dataset, or want a working reference for what a DPO training run looks like on image models. Do not adopt it as a library for production fine-tuning, since the core is a single script of more than a thousand lines with pinned 2023 era dependencies, and the project states the release is for research purposes only. Before you start, check that the pinned diffusers, transformers and accelerate versions still install on your Python, plan for a multi-GPU budget with 16 GPUs assumed by the launchers, and read the ethical considerations and AUP guidance before using the released checkpoints in anything customer facing.

## FAQ

### What is DiffusionDPO and what paper does it implement?

It is the training code for Diffusion Model Alignment Using Direct Preference Optimization, with the script adapted from the diffusers library text_to_image example. The repository publishes checkpoints for Stable Diffusion 1.5 and Stable Diffusion XL 1.0 trained as described in the paper.

### How do I install the DiffusionDPO dependencies?

The setup step is a single pip install from the requirements file, which pins accelerate 0.20.2, diffusers 0.20.0, transformers 4.30.2, xformers 0.0.22 and openai_clip 1.0.1, along with torchvision, datasets, ftfy, tensorboard, Jinja2, bitsandbytes and hpsv2.

### What hardware does running DiffusionDPO training need?

The published checkpoints are replicable with the scripts in launchers/ assuming 16 GPUs, with gradient accumulation scaled accordingly. The launcher comments also note that training takes about 24 hours over 2000 steps at an effective batch size of 2048, and Adafactor is the default optimizer for SDXL to keep memory down.

### How is a DiffusionDPO model evaluated?

The utils/ directory holds the scoring models used for evaluation or AI feedback, namely PickScore, HPS, Aesthetics and CLIP, and the model used is chosen with the --choice_model flag. The quick_samples notebook compares a pretrained model against the baseline and includes a sample of automatic quantitative evaluation using PickScore.

### Can I use the DiffusionDPO checkpoints in a product?

The project states the release is for research purposes only in support of the academic paper, that the models and code are not designed or evaluated for all downstream purposes, and that users should evaluate accuracy, safety and fairness before deploying. It points to the AUP and AI AUP for use case guidance. The code itself is Apache-2.0 licensed.

## Sources

- [Issues](https://github.com/SalesforceAIResearch/DiffusionDPO/issues)
- [License: Apache-2.0](https://github.com/SalesforceAIResearch/DiffusionDPO/blob/main/LICENSE)
- [Project website](https://arxiv.org/abs/2311.12908)
- [README](https://github.com/SalesforceAIResearch/DiffusionDPO/blob/main/README.md)
- [SalesforceAIResearch/DiffusionDPO on GitHub](https://github.com/SalesforceAIResearch/DiffusionDPO)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/salesforceairesearch-diffusiondpo