FMA-Net: joint video super-resolution and deblurring in one PyTorch pass

One model for two degradations that usually get separate networks

Video restoration pipelines are usually split by problem. A team might run a super-resolution model and then a separate deblurring model over its output, chaining two networks over the same frames. FMA-Net takes the opposite approach: the paper title given in the repository describes it as joint video super-resolution and deblurring, and the implementation is built around flow-guided dynamic filtering and iterative feature refinement with multi-attention.

The building blocks come from the vision literature rather than from anything the project invents. Optical flow is used to guide dynamic filtering, so that the filtering applied at each frame can follow motion instead of being fixed in place. Multi-attention is used to weigh the features that the refinement stage works on, and the refinement is iterative rather than a single pass.

The paper was accepted to CVPR 2024, where the news list in the repository records an oral selection on 2024-04-05 and acceptance on 2024-02-27. The code, the training and testing scripts and the pretrained model were released on 2024-04-19, and the repository itself was created on 2024-01-14. For a reader choosing a starting point, the point of this project is that it is the authors' own implementation, not a reimplementation, and the weights match the reported setup.

Two scripts and one config file drive the whole pipeline

The repository is laid out as a small research codebase rather than a packaged library. The top level holds LICENSE, README.md, an assets directory, and a handful of Python modules: config.py, data.py, main.py, model.py, train.py and utils.py, plus a preprocessing directory and a file named experiment.cfg.

That layout tells you the mechanism. main.py is the entry point and it dispatches on a mode flag, while train.py holds the training loop and data.py handles the dataset, with model.py defining the network and config.py holding defaults. The single entry point with a shared configuration file is the shape of a research script, not of a pip-installable package, and there is no setup.py or pyproject.toml in the tree. The practical implication is that you run the code from a checkout rather than importing it into a larger application.

The configuration file is the pivot. Every documented command passes --config_path experiment.cfg, so the same invocation pattern covers training, testing on the benchmark and testing on your own data. Whatever paths, options and settings your run needs are expected to travel in that file, and a change of dataset, checkpoint location or data root is a configuration edit rather than a new command line. The documentation does not walk through the contents of experiment.cfg, so expect to read it before the first run.

Getting the code and starting a training run on REDS

The documented setup is short. The requirements list Python 3.9 with PyTorch 1.9.1 or newer, and names Ubuntu 22.04 with cuda 11.8 as the platform. Clone the repository and launch training with the config file:

git clone https://github.com/KAIST-VICLab/FMA-Net
cd FMA-Net
python main.py --train --config_path experiment.cfg

The result of a successful start is a training job that reads the REDS dataset and writes checkpoints somewhere the configuration file points to. The exact output location is a configuration detail, and the documentation does not state the default paths, so the sensible first step is to open experiment.cfg and check the data root, the flow directory and the checkpoint path before launching a long run.

One environment detail deserves attention for anyone reproducing the numbers. The requirements pin an older PyTorch floor and a specific CUDA version, which is what the authors used. A newer PyTorch or a different CUDA runtime may behave differently for the same code, and the documentation offers no compatibility table, so treat the stated combination as the reference environment rather than as a minimum.

REDS4 and pseudo-GT optical flow are the data prerequisites

The model expects two things that do not ship with the repository: the REDS dataset, and optical flow for the training clips. The preprocessing section lists the REDS dataset with its download page, then gives two generation steps. The first produces the REDS4 subset by running the script in the preprocessing directory, which is the four-frame clip format the training data loader is built around. The second generates RAFT-based pseudo ground truth optical flow with a separate script, and the documentation offers a download link for that flow data as an alternative to generating it locally.

The flow detail is the one that shapes the whole approach. Because the network filters dynamically under the guidance of estimated motion, the flow files are not an optional extra for training, they are an input the design depends on. Pseudo ground truth flow means the labels come from a RAFT estimator rather than from a captured reference, which is a practical compromise for a dataset that does not ship true flow.

That also names the inference cost honestly. The model consumes motion information, so a deployment that has no flow estimate for its frames needs an estimator in the loop, and the preprocessing script in the repository is the documented way to obtain one for REDS. Whether that script is appropriate for your own footage is something the documentation does not address, and it is the first thing to check if you plan to test on custom data.

Testing on REDS and on your own clips

Two test modes are documented, and the difference between them is the mode flag rather than the configuration file. The first evaluates the model on the REDS dataset, which is the comparable setting for a paper reproduction. The second runs on your own datasets, which is the mode a practitioner would reach for:

python main.py --test --config_path experiment.cfg
python main.py --test_custom --config_path experiment.cfg

The pretrained weights come from a Dropbox link given in the repository, distributed as a zip named FMA-Net_REDS.zip and described as trained on the REDS dataset. There are no GitHub releases for this project, so the weights, the code and the flow data all come from links in the documentation rather than from a versioned release channel. That matters for reproducibility in a practical sense: there is no tag to pin, so you record the commit you cloned yourself.

Results are not tabulated in the README. Instead it points to the project page and a demo video for visual results. If you need numbers to compare against, you produce them by running the test mode on REDS with the released checkpoint, rather than by reading a table from the repository.

The licence blocks the obvious commercial use

The repository is labelled MIT, and the file list includes a LICENSE, but the licence section in the README is narrower than a standard MIT grant and it also covers the weights. The text says the source code including the checkpoint can be used freely for research and education only, and that any commercial use should get formal permission from the principal investigator, naming Prof. Munchurl Kim and the address [email protected].

This is the single most consequential fact for an adoption decision, so read the licence section in full and treat the file in the repository as the operative text if the two differ. A research group building on the method, publishing a follow-up paper, or comparing against it on a benchmark is squarely inside the stated permission. A studio that wants the model in a shipping product, an API service or a device image is outside it until a permission is in hand.

The point is not legal theory but a practical gate. The same MIT identifier that a licence scanner will report can hide a narrower grant, and a scanner is exactly the tool most teams rely on. Verify the commercial use boundary with the people who hold it before the code enters anything with revenue attached.

Where FMA-Net fits against a chained restoration pipeline

The alternative to a joint model is not a competing paper, it is the pipeline most teams already have: run super-resolution, then run deblurring over the result, and stitch the two stages together with your own orchestration code. The difference in approach is where the information is shared. In the chained case, the second network sees only the first network's output pixels, so the detail the sharpening step needs may already have been smoothed away. FMA-Net instead carries a shared feature representation through both tasks and uses optical flow to guide the filtering across time.

That argument for joint processing comes with costs that a benchmark table does not show. The model needs flow input, so the pipeline acquires a dependency that the chained approach does not have. It is a research implementation with a single entry point, a config file and no package metadata, so integrating it into a service means treating it as a repository to vendor and maintain. And the trained checkpoint is tied to REDS, the dataset the released model was trained on, so transfer to your own footage is a research question rather than a configuration change.

The honest comparison is therefore: FMA-Net is the reference implementation to build on when you are doing this research, and the wrong starting point when you need a supported, commercially usable restoration component.

Editorial conclusion

Adopt FMA-Net if you are reproducing or extending the CVPR 2024 work on the REDS benchmark, or if your own video restoration research needs a joint resolution and blur reference implementation with published weights. Do not adopt it for a commercial product without written permission, because the licence section restricts the source code and the checkpoint to research and education and requires formal permission from the principal investigator for any commercial use. Before you commit, read the licence section yourself, download FMA-Net_REDS.zip and confirm the checkpoint loads against your PyTorch version, and read experiment.cfg to see which paths it expects, since the configuration file is what points the code at the REDS data, the flow files and the model weights.

Frequently asked questions

Official sources

1. Issues
2. KAIST-VICLab/FMA-Net on GitHub
3. License: MIT
4. Project website
5. README