# diffusion-e2e-ft: turning a diffusion estimator into a single-step depth and normal predictor

> This repository holds the code and checkpoints behind a WACV 2025 oral paper on fine-tuning image-conditional diffusion models end to end on task specific losses. It suits researchers who need monocular depth or surface normals in one forward pass, and it suits them less well if they need a permissive licence or a stable pip release.

**RWTHVision/diffusion-e2e-ft** — [WACV'25 Oral] Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think

- Repository: https://github.com/RWTHVision/diffusion-e2e-ft
- Website: https://vision.rwth-aachen.de/diffusion-e2e-ft
- Stars: 523 · Forks: 22
- Language: Python
- License: not declared
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/rwthvision-diffusion-e2e-ft

## One forward pass instead of a diffusion sampler

The project behind RWTHVision/diffusion-e2e-ft asks a narrow question: how little of a diffusion model do you need to keep when the task is monocular depth estimation or monocular surface normal estimation? The answer it gives is one step. The fine-tuned models are described as single-step deterministic models, and the documentation is explicit that the noise should always be zeros while the ensemble size and the number of inference steps should always be 1. The paper was accepted as an oral at WACV 2025, with earlier news entries recording acceptance to the NeurIPS 2024 AFM Workshop. This is research code with published checkpoints rather than a packaged library, so the audience is researchers and engineers who already work with diffusion models and want the accuracy of a fine-tuned estimator without paying for sampling.

## Two starting points: a diffusion estimator or Stable Diffusion

The term E2E FT denotes models fine-tuned end to end on task specific losses, and the documentation names two possible starting points: the pretrained diffusion estimator, or Stable Diffusion directly. That distinction shows up in the checkpoint table as separate columns, and it is the interesting design decision in the repository. Starting from a diffusion estimator such as the Marigold depth checkpoint keeps the estimator's own weights and adapts them with a task loss. Starting from Stable Diffusion means the image generation prior carries the load. The table also mixes in checkpoints the authors did not train at all, and the documentation says so plainly: the Marigold depth and GeoWizard depth and normals diffusion estimators are the official checkpoints provided by their respective authors. The only estimator trained by this group, following the Marigold training regimen, is a Marigold estimator for normals.

## Zeros noise, one step, trailing timesteps

The inference argument tables are where the single-step design becomes concrete, and the defaults line up with it. For the Marigold script, denoise_steps defaults to 1 for these models, ensemble_size defaults to 1, timestep_spacing defaults to trailing for the fixed inference schedule, and noise defaults to zeros. Processing happens at a resolution of 768 by default, with 0 meaning the resolution matches the RGB input, while output_processing_res defaults to False so the generated image is resized back to the input resolution. Half precision is off by default, and resizing of both input and output uses bilinear by default. None of this is incidental. A deterministic single-step model has no sampling trajectory to spread noise along, which is why zeros is the sensible default rather than gaussian.

## Running a released checkpoint from Python

The quickstart path needs a GPU and four packages:

```bash
pip install torch diffusers transformers accelerate
```

The example loads an image and hands it to a pipeline loaded from the Hugging Face Hub with a custom pipeline argument:

```python
from diffusers import DiffusionPipeline
import diffusers

image = diffusers.utils.load_image(
    "https://gonzalomartingarcia.github.io/diffusion-e2e-ft/static/lego.jpg"
)
```

Then the depth checkpoint is loaded, moved to the GPU, run on the image, and post-processed through the image processor:

```python
pipe = DiffusionPipeline.from_pretrained(
    "GonzaloMartinGarcia/marigold-e2e-ft-depth",
    custom_pipeline="GonzaloMartinGarcia/marigold-e2e-ft-depth",
).to("cuda")
depth = pipe(image)
pipe.image_processor.visualize_depth(depth.prediction)[0].save("depth.png")
pipe.image_processor.export_depth_to_16bit_png(depth.prediction)[0].save("depth_16bit.png")
```

The same shape works for normals, using the normals checkpoint and the visualize_normals helper on the image processor, which writes normals.png. The saved prediction object carries the field named prediction in both cases.

## The Marigold script and the GeoWizard script take different arguments

Batch inference over a directory of images is handled by two separate entry points, and the argument names differ enough that copying one command into the other will not work. The Marigold script takes a checkpoint, a modality of depth or normals, an input directory and an output directory:

```bash
python Marigold/run.py \
    --checkpoint="GonzaloMartinGarcia/marigold-e2e-ft-depth" \
    --modality depth \
    --input_rgb_dir="input" \
    --output_dir="output/marigold_ft"
```

The GeoWizard script instead takes a pretrained model path, a domain and separate input and output directories:

```bash
python GeoWizard/run_infer.py \
    --pretrained_model_path="GonzaloMartinGarcia/geowizard-e2e-ft" \
    --domain indoor \
    --input_dir="input" \
    --output_dir="output/geowizard_ft"
```

The domain argument is one of indoor, outdoor or object, which means the GeoWizard route asks you to label the scene yourself. The GeoWizard table also lacks the batch_size and resample_method rows that the Marigold table lists, so do not assume the two scripts share a complete flag set.

## The alternative to fine-tuning: keep the estimator and spend steps

There is a real alternative inside this repository, and it comes from the original diffusion estimators rather than the fine-tuned checkpoints. The documentation states that by using the correct trailing timestep spacing it is possible to sample single to few-step depth maps and surface normals from diffusion estimators directly, and it adds that those samples will be blurry but become sharper by increasing the number of inference steps. So the choice is not fine-tuned or nothing. It is a one step fine-tuned model that trades sharpness for latency, or an unfine-tuned estimator where you accept blur at one step and spend more denoise steps to sharpen the result. Both routes share the same trailing timestep spacing and the same noise options, which include gaussian, pyramid and zeros.

## A research tree of pinned dependencies and no stated licence

The repository root holds the pieces of the work rather than a distributable package: a DSINE directory, a GeoWizard directory, a Marigold directory, a depth-to-normal-translator directory, an experiments directory and a training directory, alongside a requirements.txt. The pinned list is specific about versions, with torch 2.2.0, diffusers 0.30.2, transformers 4.37.2, accelerate 0.27.2, numpy 1.26.4, torchvision 0.17.0 and xformers 0.0.24, plus peft constrained below 0.14 and tooling such as wandb, datasets and omegaconf. Development is documented against Python 3.10, with a virtual environment and a requirements install as the two setup steps. The licence is the awkward part: no LICENSE file appears at the top level of the repository and no licence identifier is declared, so terms have to be established with the authors before any commercial use. The last push to the repository was on 2026-07-09, and the news list in the README stops in October 2024, so the code moves but the documentation trail is old.

## Conclusion

Adopt diffusion-e2e-ft if you want monocular depth or surface normals from a deterministic single step model and you are willing to work from source with pinned dependencies and a GPU. Do not adopt it if you need a released package version or a clearly stated licence, because the repository publishes no GitHub releases and its root contains no LICENSE file. Before you build on it, run the depth checkpoint from the Marigold script with the default single step settings and confirm the output resolution behaviour, since processing_res defaults to 768 and the generated map is resized to the input by default.

## FAQ

### How do I run inference with the diffusion-e2e-ft models?

Load a checkpoint through DiffusionPipeline.from_pretrained with the matching custom_pipeline value and move it to cuda, then pass the image to the pipeline, or use the Marigold run.py script for batch inference over a directory with the checkpoint, modality, input_rgb_dir and output_dir arguments.

### Which checkpoints are available in diffusion-e2e-ft?

The table lists stable-diffusion-e2e-ft-depth and marigold-e2e-ft-depth for depth, stable-diffusion-e2e-ft-normals and marigold-e2e-ft-normals for normals, and geowizard-e2e-ft for depth and normals together. The Marigold depth and GeoWizard estimators are the official checkpoints from their authors, and the Marigold normals estimator is the one trained by this group.

### What denoise_steps, ensemble_size and noise should I use with the fine-tuned models?

The documentation states that because the fine-tuned models are single-step deterministic models, the noise should always be zeros and the ensemble size and number of inference steps should always be 1. Those values are also the defaults in the argument tables.

### What does E2E FT mean in this repository?

It denotes models fine-tuned end to end on task specific losses, starting either from the pretrained diffusion estimator or directly from Stable Diffusion. The starting point is what separates the checkpoints in the model table from each other.

### Can I use diffusion-e2e-ft code in a commercial product?

That needs checking with the authors. No LICENSE file appears among the top level entries of the repository and no licence identifier is declared, so the terms are not stated in the project itself.

## Sources

- [Issues](https://github.com/RWTHVision/diffusion-e2e-ft/issues)
- [Project website](https://vision.rwth-aachen.de/diffusion-e2e-ft)
- [README](https://github.com/RWTHVision/diffusion-e2e-ft/blob/main/README.md)
- [RWTHVision/diffusion-e2e-ft on GitHub](https://github.com/RWTHVision/diffusion-e2e-ft)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/rwthvision-diffusion-e2e-ft