# CDDFuse: correlation-driven fusion for infrared and medical image pairs

> CDDFuse is the official implementation of a CVPR 2023 paper on multi-modality image fusion, and its idea is decomposition: a dual branch extractor pulls global low-frequency structure and local high-frequency detail apart, and a correlation driven loss keeps the two apart in the right way. The repository ships training code, two pretrained weights and the metric tables from the paper, pinned to a 2022 era Python stack.

**Zhaozixiang1228/MMIF-CDDFuse** — [CVPR 2023] Official implementation for "CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion."

- Repository: https://github.com/Zhaozixiang1228/MMIF-CDDFuse
- Stars: 639 · Forks: 58
- Language: Python
- License: not declared
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/zhaozixiang1228-mmif-cddfuse

## The idea is to split the image before you combine it

Multi-modality image fusion aims to render fused images that maintain the merits of different modalities, for example the functional highlight of an infrared image and the detailed texture of a visible one. The hard part, as the paper puts it, is modelling cross-modality features and decomposing desirable modality specific and modality shared features.

CDDFuse, which stands for Correlation-Driven feature Decomposition Fusion, is the answer to that. The pipeline has a recognisable shape. Restormer blocks extract cross-modality shallow features. Then a dual-branch Transformer-CNN feature extractor does the decomposition, with Lite Transformer blocks using long-range attention to handle low-frequency global features and Invertible Neural Network blocks focusing on high-frequency local information.

The loss is where the correlation comes in. A correlation-driven loss is proposed to make the low-frequency features correlated while the high-frequency features stay uncorrelated, based on the embedded information. In other words, the network is trained so that the global structure agrees between the two modalities while the local detail does not, which is exactly the property that makes a fused image look right.

The two branches then do their own fusions, an LT-based global fusion layer and an INN-based local fusion layer, and their outputs combine into the fused image.

The paper also reports downstream gains, showing that CDDFuse can boost performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark.

## A repository that is really the paper, with a runnable appendix

This is the official implementation for CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion, published at CVPR 2023, and the README is arranged as a citation package rather than as a product page.

The authors are Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin, Radu Timofte and Luc Van Gool, with a project page, the paper in the CVF open access archive, the arXiv entry at 2104.06977 and the supplementary materials linked alongside. A BibTeX block is given for the inproceedings entry, with the page range 5906-5916 and the June 2023 date.

The update log shows how the code was released in two steps. In April 2023 the inference code for infrared-visible image fusion and medical image fusion was released, and in June 2023 the training codes and config files became public.

That order matters for anyone planning to use the project. The inference path was published first and is the part with pretrained weights, so a user who only wants to fuse images can start there, while anyone training from scratch needs the later release.

The repository top level is small and flat, which is typical of a released research code base. There is net.py for the network architecture, dataprocessing.py, train.py, test_IVF.py and test_MIF.py for the two tasks, a utils/ directory, the models/ directory for weights, test_img/ for the test data, an image/ directory and the MSRS_train/ directory. There is no package metadata, no setup script and no continuous integration configuration in the listing.

## Training from scratch on the MSRS dataset

The training section is a four step sequence, and it starts with the environment.

The virtual environment is created with conda at Python 3.8.10, and the requirements are installed from the file:

```bash
# create virtual environment
conda create -n cddfuse python=3.8.10
conda activate cddfuse
# select pytorch version yourself
# install cddfuse requirements
pip install -r requirements.txt
```

The comment about selecting the PyTorch version yourself matters, and it is repeated in the requirements file. PyTorch is not pinned there because the version depends on whether you are on CUDA or CPU, so it is your one manual decision.

The data is the MSRS dataset, which you download from the Linfeng-Tang/MSRS repository and place in a folder called MSRS_train. Then the preprocessing step turns the raw pairs into a single HDF5 file:

```bash
python dataprocessing.py
```

and the processed training dataset lands in a file named MSRS_train_imgsize_128_stride_200.h5, which tells you the crop size and the stride used when building the patches. Training is then a single command:

```bash
python train.py
```

with the trained model ending up in the models/ directory.

The parameters in that file name are the interesting part for reproduction, since they are the only description of the training configuration in the repository, and the config files themselves are referenced in the June 2023 update without being reproduced in the README.

## Two pretrained weights and five test sets

For anyone who does not want to train, the repository ships two pretrained models. models/CDDFuse_IVF.pth handles the Infrared-Visible Fusion task and models/CDDFuse_MIF.pth handles Medical Image Fusion.

The test data is organised by task. For infrared-visible fusion there are the RoadScene and TNO sets under test_img/, and for medical image fusion there are MRI_CT, MRI_PET and MRI_SPECT.

One constraint is stated plainly, and it is a real one. Since the size of the MSRS dataset for IVF is 500+MB, the authors could not upload it for exhibition, so it has to be downloaded separately from the Linfeng-Tang/MSRS repository. The other datasets contain all the test images.

That means a clone of the repository gives you everything for evaluation and nothing for training, which is a sensible split but one that trips people up when the training script cannot find its data.

Inference is a single script per task, with test_IVF.py for infrared-visible fusion and test_MIF.py for medical image fusion, and the results are printed in the terminal. The README states that running them reproduces the numbers in the original paper, matching Table 1 for infrared-visible fusion and Table 5 for medical image fusion.

## What the printed metrics actually tell you

The terminal output is a metric table, and reading it is the fastest way to judge whether a run matches the paper. The columns are EN, SD, SF, MI, SCD, VIF, Qabf and SSIM, eight standard fusion metrics evaluated per dataset.

For the infrared-visible task, the TNO set prints EN 7.12, SD 46.0, SF 13.15, MI 2.19, SCD 1.76, VIF 0.77, Qabf 0.54 and SSIM 1.03, and RoadScene prints EN 7.44, SD 54.67, SF 16.36, MI 2.3, SCD 1.81, VIF 0.69, Qabf 0.52 and SSIM 0.98.

The medical output is more interesting, because it prints two rows per dataset, CDDFuse_IVF and CDDFuse_MIF, on the same data. For MRI_CT, the IVF model gives EN 4.83, SD 88.59, SF 33.83, MI 2.24, SCD 1.74, VIF 0.5, Qabf 0.59 and SSIM 1.31, while the MIF model gives EN 4.88, SD 79.17, SF 38.14, MI 2.61, SCD 1.41, VIF 0.61, Qabf 0.68 and SSIM 1.34.

The pattern repeats on MRI_PET and MRI_SPECT, with the MIF model generally showing higher MI, SF, VIF, Qabf and SSIM and lower SD and SCD. So the second column set is a cross-task comparison, the MIF checkpoint applied to medical pairs against the IVF checkpoint applied to the same pairs, and the differences are not one-directional, which is a reasonable thing to see before you trust a checkpoint on data it was not trained for.

## The pinned dependency list is the real constraint

The requirements file is the most consequential document in the repository, and it is fully pinned:

```
einops==0.4.1
h5py==3.7.0
kornia==0.2.0
numpy==1.21.5
opencv_python==4.5.3.56
scikit_image==0.19.2
scikit_learn==1.1.3
scipy==1.7.3
tensorboardX==2.5.1
tqdm==4.64.1
# Install PyTorch separately for your CUDA/CPU setup.
```

The science stack is einops for tensor rearrangement, h5py for the preprocessed dataset, kornia for image operations, numpy, opencv, scikit-image, scikit-learn, scipy, tensorboardX for training logs and tqdm for progress bars, and PyTorch is installed separately for your CUDA or CPU setup.

Two of those pins are going to be friction on a current machine. numpy 1.21.5 and scikit-learn 1.1.3 are from the 2022 era, and the environment is specified as Python 3.8.10, so the combination expects an older interpreter. Reproducing the paper numbers means matching that stack, and anyone who tries to install the pins into a modern Python will hit the wall before the code.

That is the honest trade-off of a research release, and the mitigation is straightforward: build the environment the README describes, or build a current one and accept that you are no longer reproducing the paper exactly. The second option is a research decision, not an installation problem.

## Where this fits in a modern pipeline

CDDFuse is a paper implementation, and there are two ways that can go wrong when you adopt one.

The first is the environment, which the previous section covers. The second is scope, and it is easier to get wrong because the results look good. The repository demonstrates fusion quality and, in the paper, a benefit to downstream infrared-visible semantic segmentation and object detection in a unified benchmark. It does not ship a segmentation or detection pipeline, so the downstream claim is something you would have to reproduce in your own codebase.

The alternative approaches differ in kind. A general purpose image processing library gives you classical fusion, weighted averaging and gradient based methods, which are cheap, deterministic and need no model weights, but which do not learn what makes an infrared highlight worth keeping. A general purpose deep fusion model gives you a single network trained for a specific pair of modalities on a specific dataset, which is what CDDFuse is, with the difference that its contribution is the explicit decomposition of shared and modality specific features rather than the fusion itself.

The decomposition is also the part that will matter if you extend it. Because the low-frequency and high-frequency paths are trained to be correlated and uncorrelated respectively, a downstream user can in principle take the shared branch alone as a structure estimate or the specific branch as a modality mask, which is the kind of property a general purpose fusion model does not offer.

On maintenance, the last push was on 2026-06-08, the repository has no GitHub releases, and the licence is not stated. That combination means you should treat the code as a reference implementation tied to the paper rather than as a maintained library, and if you need a supported dependency, you are reading a research release.

## Conclusion

Adopt CDDFuse if you need infrared and visible pairs or medical image pairs fused with an explicit decomposition of shared and modality specific content, and you are willing to work inside the pinned dependency set the authors used. Skip it if you want current tooling, since the requirements file pins numpy 1.21.5 and Python 3.8.10, or if you need a maintained release path, because the repository has no releases and the licence is not stated. Before you run anything, read the metric table the test scripts print and compare it with the numbers in the paper, install PyTorch separately for your CUDA or CPU setup as the requirements file instructs, and note that the MSRS training set has to be fetched separately because it is too large to upload. The last push was on 2026-06-08.

## FAQ

### How do I run CDDFuse inference on infrared and visible image pairs?

Use the pretrained weight models/CDDFuse_IVF.pth and run python test_IVF.py, which prints the metric table for the RoadScene and TNO test sets. For medical pairs, use models/CDDFuse_MIF.pth and run python test_MIF.py.

### What are the main components of the CDDFuse architecture?

Restormer blocks extract cross-modality shallow features, a dual-branch Transformer-CNN extractor uses Lite Transformer blocks for long-range attention on low-frequency global features and INN blocks for high-frequency local information, and a correlation-driven loss keeps the low-frequency features correlated while the high-frequency features stay uncorrelated.

### Why can I not find the MSRS dataset in the repository?

The README states that the MSRS dataset for infrared-visible fusion is 500+MB and could not be uploaded for exhibition, so it has to be downloaded from the Linfeng-Tang/MSRS repository and placed in the MSRS_train folder. The other test datasets are included.

### What Python and dependency versions does CDDFuse expect?

The README creates the environment with conda at python=3.8.10, and the requirements file pins numpy 1.21.5, scipy 1.7.3, scikit-learn 1.1.3, h5py 3.7.0 and kornia 0.2.0, with a note that PyTorch is installed separately for your CUDA or CPU setup.

## Sources

- [Issues](https://github.com/Zhaozixiang1228/MMIF-CDDFuse/issues)
- [README](https://github.com/Zhaozixiang1228/MMIF-CDDFuse/blob/main/README.md)
- [Zhaozixiang1228/MMIF-CDDFuse on GitHub](https://github.com/Zhaozixiang1228/MMIF-CDDFuse)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/zhaozixiang1228-mmif-cddfuse