Custom Diffusion: fine-tuning Stable Diffusion on a few images with a 75MB delta
Custom Diffusion: Multi-Concept Customization of Text-to-Image Diffusion (CVPR 2023)
At a glance
- What is it?
- Custom Diffusion trains a small set of cross-attention key and value weights so a text-to-image model can learn a new subject from roughly 4 to 20 images, and it can combine several learned concepts in one prompt. The repository is a research codebase tied to a specific Stable Diffusion commit, and the README now points most users at the diffusers implementation.
- Who is it for?
- Adopt Custom Diffusion if you need to teach one diffusion model several distinct subjects and want each concept to stay a small, swappable file rather than a full model copy. Do not adopt it if you want a maintained training pipeline or a clean license story: the repository is a research snapshot whose README now routes training and inference through the diffusers example, and the license file is not a standard identifier.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 116 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 18, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Custom Diffusion targets: many subjects, one model
Text-to-image models generate whatever the training data taught them. Getting one to draw your specific object, pet or style normally means either prompting around the gap or retraining a large portion of the model per subject. Custom Diffusion takes the second path but narrows it: given a few user images of a concept, roughly 4 to 20 according to the README, it augments a pre-trained diffusion model so the concept appears in unseen contexts.
The audience is narrow. This is for people who want to compose concepts, not just reproduce one. The README lists the combinations it supports: new object plus new artistic style, multiple new objects, and new object plus new category. That compositional goal is what separates it from single-subject personalization. If you only ever need one subject, the multi-concept machinery is overhead you will not use.
How the fine-tuning works: key and value matrices, not the whole model
The mechanism is parameter selection. Custom Diffusion fine-tunes only the key and value projection matrices in the cross-attention layers, the mapping from text to latent features. Everything else in the diffusion model stays frozen. The README states the training run takes about 6 minutes on 2 A100 GPUs and that the extra storage per concept is 75MB, both direct consequences of touching such a small slice of the weights.
Two details shape daily use. First, a small set of 200 regularization images is used to prevent overfitting, and the repository ships both real-image and generated-image regularization paths. Second, for personal categories the method introduces a modifier token V* in front of the category name, written in the examples as a placeholder like <new1> cat. When you train several concepts jointly, each gets its own index, and prompts reference them by number.
The repository also describes merging two fine-tuned models through an optimization step rather than by simply concatenating their deltas. That is the part of the design that makes the multi-concept claim credible: the deltas are small enough to store separately and combine at sampling time.
Installing Custom Diffusion from the repository
The README's Getting Started block clones this repository, then clones the CompVis stable-diffusion repository inside it, then creates a conda environment from that repository's environment.yaml. The project was developed against a specific stable-diffusion commit, 21f890f9da3cfbeaba8e2ac3c425ee9e998d5229, so a fresh clone of the default branch is not the tested configuration.
git clone https://github.com/adobe-research/custom-diffusion.git
cd custom-diffusion
git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion
conda env create -f environment.yaml
conda activate ldm
pip install clip-retrieval tqdmThe environment is named ldm. After activating it, the README downloads the stable-diffusion-v1-4 checkpoint, which the training and sampling scripts expect as a path argument.
wget https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckptFor a first real run, the single-concept flow with real images as regularization is the shortest path. It downloads the released dataset, unzips it, and calls the finetune script with a concept name, its data folder, a regularization folder, an output name, the config file and the checkpoint path. The README annotates this step as needing 30 GB on 2 GPUs.
wget https://huggingface.co/datasets/nupurkmr9/custom-diffusion/resolve/main/data.zip
unzip data.zip
bash scripts/finetune_real.sh "cat" data/cat real_reg/samples_cat cat finetune_addtoken.yaml <pretrained-model-path>Training writes into a logs folder. The next step extracts the delta weights, and the --newtoken 1 flag corresponds to the single added token for this run. You then sample with the extracted delta rather than the full model.
python src/get_deltas.py --path logs/<folder-name> --newtoken 1
python sample.py --prompt "<new1> cat playing with a ball" --delta_ckpt logs/<folder-name>/checkpoints/delta_epoch\=000004.ckpt --ckpt <pretrained-model-path>What you should see is a generated image of the learned cat in a new scene. If the prompt contains <new1> but the delta is not loaded, the token is meaningless to the base model, which is the fastest way to confirm whether your checkpoint path is correct.
Multi-concept training and the joint-training script
The multi-concept path is a second script with a different argument shape. The README's example trains a wooden pot and a cat together, passing each concept's name, data folder and regularization folder, then a combined output name, a joint config file and the checkpoint.
bash scripts/finetune_joint.sh "wooden pot" data/wooden_pot real_reg/samples_wooden_pot \
"cat" data/cat real_reg/samples_cat \
wooden_pot+cat finetune_joint.yaml <pretrained-model-path>Because two tokens are added, the delta extraction uses --newtoken 2. Sampling then references both tokens in one prompt, and the README's example sentence places them in different grammatical roles to show the composition.
python src/get_deltas.py --path logs/<folder-name> --newtoken 2
python sample.py --prompt "the <new2> cat sculpture in the style of a <new1> wooden pot" --delta_ckpt logs/<folder-name>/checkpoints/delta_epoch\=000004.ckpt --ckpt <pretrained-model-path>The ordering convention matters. Token numbering follows the order in which concepts were passed to the script, so a reader who reorders the arguments and keeps the old prompt will get swapped results. The README does not document a validation check for this.
Where Custom Diffusion is the wrong tool
The most concrete limitation is the dependency pin. The README states the code was developed on stable-diffusion commit 21f890f9da3cfbeaba8e2ac3c425ee9e998d5229. Nothing in the repository promises compatibility with later versions of that codebase, and the environment is built from its environment.yaml rather than from a lockfile owned by this project.
The second limitation is memory. The README labels the training command as 30 GB on 2 GPUs. That is not a consumer-hardware budget, and the repository does not describe a reduced-memory path.
The third is scope. Custom Diffusion learns concepts, and the README frames the results around fine-tuning stable-diffusion-v1-4. It is not a general image editing tool, not a way to improve prompt adherence on concepts the base model already knows, and not a replacement for a full fine-tune when you want the model's overall behaviour to shift. The README also notes that the paper's results did not use clip-retrieval for gathering real regularization images, even though the install step includes clip-retrieval and the script accepts a real_reg folder. Generated-image regularization is offered as the alternative script.
Custom Diffusion compared with DreamBooth-style fine-tuning
The natural comparison is DreamBooth, which the related searches surface alongside this project. The difference is where the learning happens. A DreamBooth-style run fine-tunes the diffusion model itself, so the resulting artifact is a full model or a large weight update. Custom Diffusion freezes the model and trains only the key and value projections in cross-attention, which is why the README can quote 75MB per concept and about 6 minutes on 2 A100 GPUs.
That difference changes the workflow. In the Custom Diffusion flow you keep one base checkpoint and a folder of small delta files, then select which delta to load at sampling time via --delta_ckpt. Combining concepts is a matter of training jointly or merging two deltas through the optimization step the README mentions. The trade-off is that the learned signal lives in a thin slice of the network, so the method is optimized for subject and style identity rather than for broad changes in how the model behaves.
The README also points elsewhere for the modern path: Custom Diffusion is supported in diffusers, and the README refers readers to the diffusers custom_diffusion example for training and inference details, including an SDXL variant with diffusers==0.21.4. For most new work that is the entry point worth reading first, because it does not require reproducing this repository's pinned stable-diffusion checkout.
Maintenance, licensing and upgrade cost
The repository is not archived, and the last push was on 2026-05-24. That is recent enough that the code has not been abandoned, but the README's own framing is that the supported path has moved to diffusers. Treat this repository as the reference implementation for the paper and the source of the released dataset and models, not as the pipeline you build a product on.
Upgrade cost concentrates in two places. The stable-diffusion dependency is pinned by commit, so any upstream change requires you to re-validate training and sampling yourself. The second is the delta format: checkpoints are extracted per token count with --newtoken, so a change in how many concepts you train changes the extraction step, not just the prompt.
On licensing, the repository contains both LICENSE.md and MIT_License.md, and the license is reported as NOASSERTION, meaning no standard identifier was detected. The README separately notes that images taken from UnSplash are under the UnSplash License, and that the stable-diffusion-v1-4 checkpoint is downloaded from its own Hugging Face repository with its own terms. Those are three separate sets of terms for code, dataset images and model weights. This is not legal advice; if you plan to ship generated output, read each of those files rather than assuming the MIT file covers everything.
Editorial conclusion
Adopt Custom Diffusion if you need to teach one diffusion model several distinct subjects and want each concept to stay a small, swappable file rather than a full model copy. Do not adopt it if you want a maintained training pipeline or a clean license story: the repository is a research snapshot whose README now routes training and inference through the diffusers example, and the license file is not a standard identifier. Verify first that your GPU memory covers the documented 30 GB across 2 GPUs, that you can check out stable-diffusion commit 21f890f9da3cfbeaba8e2ac3c425ee9e998d5229, and that the delta_epoch=000004.ckpt file your run produces loads through sample.py.
Frequently asked questions
What are the three types of diffusion?
The README does not describe a taxonomy of diffusion types. It covers one use of diffusion models: fine-tuning a pre-trained text-to-image model such as Stable Diffusion by training only the key and value projection matrices in the cross-attention layers.
What exactly is a diffusion model?
The README treats diffusion models as pre-trained text-to-image generators that can be fine-tuned, naming Stable Diffusion and its stable-diffusion-v1-4 checkpoint as the base. It does not define the underlying method beyond the cross-attention layers it modifies.
Is ChatGPT a diffusion model?
The README does not mention ChatGPT. Everything it describes concerns text-to-image diffusion models, specifically fine-tuning stable-diffusion-v1-4 to learn new concepts from a few images.
What is AI diffusion?
The README does not give a general definition of AI diffusion. It describes a specific application: augmenting a pre-trained text-to-image diffusion model with a new concept using roughly 4 to 20 user images, a modifier token such as <new1>, and 200 regularization images.
Community notes