flownet2-pytorch: NVIDIA's PyTorch Port of FlowNet 2.0, and How to Run It

What FlowNet 2.0 Solves and Who Needs It in PyTorch

Optical flow estimates the per-pixel motion between two consecutive frames. FlowNet 2.0 stacked several smaller FlowNet variants so that a second network refines the flow produced by the first, which improved accuracy on the MPI-Sintel benchmark over the original single-network design. The original release was Caffe. This repository is the PyTorch port, and it exists so that people who train and deploy in PyTorch do not have to maintain a Caffe toolchain to run or fine-tune the architecture.

The README lists six architectures: FlowNet2S, FlowNet2C, FlowNet2CS, FlowNet2CSS, FlowNet2SD and FlowNet2, each with a batchnorm variant. That range matters because the stacked models are not interchangeable. FlowNet2 is the full stack and the slowest; the S and C variants are the building blocks. If you are benchmarking against a modern estimator, you want FlowNet2. If you are ablating how much the stacking contributes, you want the smaller ones.

The audience is narrow and specific: researchers reproducing a 2017 CVPR result, engineers who need a flow baseline for a comparison table, and teams fine-tuning on domain data where a general-purpose model trained on FlyingChairs and FlyingThings does not transfer. It is not a library for people who want a flow number from a webcam stream without reading a paper.

Custom CUDA Layers: Correlation and Resample2d

The distinguishing part of the codebase is not the network definitions, which are ordinary Python in networks/. It is the two custom layers. FlowNet2 and the FlowNet2C family depend on Resample2d and Correlation, and the repository ships CUDA kernels for both. Correlation computes the cost volume that lets the network compare features across a displacement range; Resample2d warps a feature map by a flow field. Neither exists in stock PyTorch, which is why install.sh has to compile them.

The README states plainly that half-precision kernels are not available for these layers. That is a real constraint, not a footnote. The project advertises fp16 inference support, but the two layers that make FlowNet2 what it is will fall back or fail in half precision. Anyone planning to use fp16 to fit the 620MB FlowNet2 model into a smaller GPU budget should read that line twice: the speedup applies to the parts of the graph that are not these layers.

Installing the kernels is also where portability breaks down. The Dockerfile pins FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04. CUDA 9.0 is old, and the nvcc invocation in install.sh is not documented in the README beyond the single bash command. Expect to debug compiler flags if you are not on that base image.

How the Data Flows from Image Pairs to Predicted Flow

The entry point is main.py, and it handles both training and inference behind one argument parser. Datasets come from datasets.py, which provides loaders for FlyingChairs, FlyingThings, ChairsSDHom and a generic ImagesFromFolder. Losses come from losses.py, which offers L1 and L2 with multi-scale support. The multi-scale loss is what the second training example uses, with --loss_numScales=5 and --loss_startScale=4, meaning the loss is computed at five scales starting from a coarse one.

Inference takes a dataset name, a dataset root and a checkpoint. The README's example uses --inference_dataset MpiSintelClean with --save_flow, which writes flow output alongside the input. Training takes the same shape: a training dataset, a validation dataset, a model name, an optimizer and a learning rate. The design is deliberately uniform, so swapping MPI-Sintel for your own folder means pointing --training_dataset_root at a directory the ImagesFromFolder loader understands.

There is no configuration file and no Python API documented for embedding the model in a larger program. Everything runs through main.py. If you need flow inside a service, you will be reading models.py and constructing the network yourself, and the README does not describe that path.

Installing flownet2-pytorch and Running a First Inference

The README gives two install steps: clone the repository, then run install.sh, which compiles the custom layers. Python 3 is required, and the pinned PyTorch version is 0.4.1. The README points readers on 0.4.0 or older to the python36-PyTorch0.4 branch. The remaining requirements are numpy, scipy, scikit-image, tensorboardX, colorama, tqdm and setproctitle.

git clone https://github.com/NVIDIA/flownet2-pytorch.git
cd flownet2-pytorch
bash install.sh

After install.sh finishes, the CUDA extensions for Correlation and Resample2d should be importable. If the build fails, the error will come from nvcc rather than from Python, and the README offers no troubleshooting section for that.

Before inference you need weights. The README lists converted Caffe pre-trained models on Google Drive, including FlowNet2 at 620MB and the smaller variants from 148MB upward. Download the one matching your --model argument and point --resume at the checkpoint directory.

python main.py --inference --model FlowNet2 --save_flow --inference_dataset MpiSintelClean \
--inference_dataset_root /path/to/mpi-sintel/clean/dataset \
--resume /path/to/checkpoints

With --save_flow set, the run writes predicted flow for each pair in the dataset root. For training on MPI-Sintel Final with validation on Clean, the README's first example sets batch size 8, Adam at 1e-4, and L1Loss.

python main.py --batch_size 8 --model FlowNet2 --loss=L1Loss --optimizer=Adam --optimizer_lr=1e-4 \
--training_dataset MpiSintelFinal --training_dataset_root /path/to/mpi-sintel/final/dataset  \
--validation_dataset MpiSintelClean --validation_dataset_root /path/to/mpi-sintel/clean/dataset

The README notes that the same commands work for other datasets, and that python main.py --help prints the full argument list. Multiple GPU training is supported, though the README does not specify the flag that enables it.

Where flownet2-pytorch Breaks or Is the Wrong Choice

The PyTorch 0.4.1 pin is the largest practical problem. PyTorch has changed its extension build system, its autograd internals and its CUDA packaging since 0.4.1. A repository that requires it will not run against a current PyTorch install without work, and the README does not document a migration path beyond the older branch for 0.4.0 and below. If your environment is standardized on a recent PyTorch, this project is the wrong tool unless you are prepared to maintain a separate environment for it.

The missing half-precision kernels for Correlation and Resample2d are the second issue. The README advertises fp16 inference, then states the custom layers have no half-precision kernels. That combination means the advertised speedup is partial, and the two heaviest operations may be excluded from it.

Third, the pre-trained weights are not covered by the repository licence. The README says that using the converted Caffe models requires adhering to a separate licence agreement hosted on Google Drive. The repository's own LICENSE file is present at the top level, but the GitHub API reports the licence as NOASSERTION, so the exact terms are not machine-readable from the repository metadata. Anyone shipping a product that uses these weights needs to read that separate agreement rather than assume the code licence covers them.

Finally, accuracy. FlowNet 2.0 is a 2017 result. If your goal is the best flow on a modern benchmark rather than reproducing a specific paper, a newer architecture will serve you better, and this repository's value is as a faithful PyTorch baseline.

Alternatives: RAFT, PWC-Net and GMFlow

The README itself points to PWC-Net from Nvidia, available in both Caffe and PyTorch. The difference in approach is architectural: PWC-Net builds a feature pyramid and warps features at each level, using a cost volume computed on warped features, while FlowNet 2.0 stacks whole networks so each refines the previous one's output. PWC-Net is a single network with a coarse-to-fine loop; FlowNet 2.0 is a cascade of separate models, which is why its full variant is 620MB and slow.

RAFT takes a different route again. It computes a dense correlation volume once and then iteratively updates the flow field with a recurrent unit, rather than predicting flow in one pass or cascading networks. In practice RAFT and GMFlow are the names that come up when people search for modern optical flow code, and both are commonly used as the comparison point against FlowNet 2.0 in papers. GMFlow reformulates flow as a global matching problem instead of a local correlation search.

If you need FlowNet 2.0 specifically, none of these substitutes for it, because the point of using it is the specific published result. If you need accurate flow and do not care which architecture produces it, the newer options are the ones to evaluate first, and this repository's role becomes that of a baseline you cite rather than a component you deploy.

Maintenance, Licence and the Cost of Keeping It Running

The repository is not archived, and the last push was on 2026-03-30. The repository has no published GitHub releases, so there is no tagged version to pin against; you are tracking the master branch. The commit history is not described in the README, so how much of that recent activity is substantive versus housekeeping cannot be determined from the repository metadata.

Upgrade cost is dominated by the environment. The Dockerfile bases everything on nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 and installs torch 1.0.0 from the cu90 wheel index in its Python layer, which is a different version from the 0.4.1 the README requires. That inconsistency between the Dockerfile and the README is worth resolving before you build on either. launch_docker.sh is present at the top level for running the container.

On licensing, the code and the weights are governed separately. The README directs users of the converted Caffe pre-trained models to a separate licence agreement, and the repository metadata does not assert a standard licence identifier. The safest reading is that the code terms come from the LICENSE file and the weight terms come from that agreement, and that they may not be the same. This is not legal advice; it is a pointer to the two documents you would need to read.

Editorial conclusion

Adopt flownet2-pytorch if you need to reproduce FlowNet 2.0 results in PyTorch, fine-tune the stacked variants on your own image pairs, or run the supplied Caffe weights as a baseline against newer estimators such as RAFT or GMFlow. Do not adopt it for a new production pipeline: the README pins PyTorch 0.4.1, the custom CUDA kernels have no half-precision path, and the pre-trained weights carry a separate licence. Verify first that your CUDA toolchain matches the Dockerfile's cuda:9.0-cudnn7 base, that install.sh builds Resample2d and Correlation on your machine, and that a single inference run on MPI-Sintel produces the flow files you expect before committing to a training job.

Frequently asked questions

Official sources

1. Issues
2. NVIDIA/flownet2-pytorch on GitHub
3. README