Fast-SRGAN trades a little quality for 30fps upscaling

Pixel shuffle is the change that makes it fast

The stated goal of the repository is to enable real time super resolution for upsampling low resolution videos, and the design follows the SR-GAN architecture with one substitution. For speed, the upsampling is done through pixel shuffle. That single design decision explains the rest of the project. The original SR-GAN upsamples with transposed convolutions, which is expressive and expensive, while pixel shuffle rearranges channels that were computed at low resolution into a higher resolution grid, which is far cheaper in both memory and arithmetic. The repository is a TensorFlow 2 implementation, and the topic list reflects the ingredients, with cnn, gans, generative adversarial network, residual blocks, single image super resolution, real time super resolution and tensorboard. The repository is small and readable, with model.py holding the network, dataloader.py the data pipeline, trainer.py the training loop, train.py and inference.py the entry points, and a configs/ directory holding the configuration.

The published numbers, and what they were measured on

The README gives a speed table, and the conditions matter as much as the values. The runtimes and frames per second are obtained by averaging runtimes over 800 frames, measured on MPS on a MacBook M1 Pro GPU. At an input of 90x160 producing 360x640, or 360p output, the time is 0.01 seconds and the rate is 82 frames per second. At an input of 180x320 producing 720x1080, or 720p output, the time is 0.04 seconds and the rate is 27 frames per second. The conclusion the project draws is that it is possible to upsample to 720p at around 30fps. Two caveats are worth stating. These are the project's own measurements on one specific Apple GPU, so the numbers on an NVIDIA card or a CPU will differ, and no fidelity metric is reported here, so you know the speed and not the quality. The samples section gives a visual comparison instead, with the low resolution image upsampled 4x by bicubic on the left, the model output in the middle and the actual high resolution image on the right.

Installing, and the pretrained generator

The requirements section says the project was tested on Python 3.10, and the install uses a Pipfile rather than a hand written requirements list:

pip install pipenv --upgrade
pipenv install --system --deploy

A pretrained generator model trained on the DIV2K dataset is provided in the models directory, and it uses 8 residual blocks with 64 filters in every layer of the generator. To try it on your own images, the inference script takes an input directory and an output directory:

python inference.py --image_dir 'path/to/your/image/directory' --output_dir 'path/to/save/super/resolution/images'

The repository ships a Pipfile and a Pipfile.lock, so the dependency set is pinned, which matters for a project whose performance claim is hardware dependent. The inference entry point is directory based rather than a single file argument, which is the shape you want for a batch of frames and is also the reason you would preprocess a video into images before running it.

Training is configured with Hydra, so the CLI is the interface

Training starts from a YAML config in configs/config.yaml, and the entry point is:

python train.py

The same parameters can be overridden from the command line, and the README gives an example that runs training with a batch size of 32, a generator with 12 residual blocks and a path to the image directory:

python train.py data.image_dir="/path/to/image/dataset" training.batch_size=32 generator.n_layers=12

This is powered by Hydra, which means all the parameters in the config are editable via the CLI, and the practical consequence is that the config file is a source of defaults rather than a thing you have to edit for every experiment. The residual block count is the interesting knob, since the pretrained model uses 8 and the example uses 12, and more blocks means more capacity and more compute. Checkpoints and training summaries are saved in tensorboard, and to monitor training progress you open tensorboard pointed at the outputs directory that is created when training starts. The architecture diagram in the README shows the training setup with the discriminator and the generator, and the project follows the SR-GAN design it references.

What to check before you build on it

A few things are worth settling before you commit to this repository. The performance figures are tied to MPS on an M1 Pro, so if your deployment runs on an NVIDIA GPU or on CPU, you need to measure for yourself, and the README does not claim a result for those. The scale factor is fixed by the pretrained generator, and the comparison samples use a 4x factor, so if your source and target resolutions are not related by that factor you are retraining rather than running inference. The directory based inference means you need frames on disk, which adds a decode and encode step to a video pipeline, and a real time video application would want a frame level interface instead. There is also no published fidelity measurement in the README, and the contributing section invites pull requests for improving model performance or adding metrics, which confirms that metrics were not the focus of the work. The MIT licence and the small, readable source make it a good base to adapt, which is a reasonable way to treat the project given what it is.

Editorial conclusion

Adopt Fast-SRGAN when you have a real time video pipeline and want the SR-GAN architecture at a frame rate that works for playback, and when a Mac GPU or similar hardware is where it will run. Do not adopt it expecting the quality of a slower GAN, since the speed comes from changing the upsampling and the block count, and the README does not report fidelity metrics. Verify first which hardware you have, since the published figures were measured on MPS, that the scale factor of the pretrained generator matches the factor you need, and that the inference script's directory mode fits how you feed frames in.

Frequently asked questions

Official sources

1. HasnainRaz/Fast-SRGAN on GitHub
2. Issues
3. License: MIT
4. README