DPVO: deep patch visual odometry and SLAM in one repository

Two papers in one repository

The repository contains the source code for two papers, and the citations are in the README so you know which is which.

Deep Patch Visual Odometry is by Teed, Lipson and Deng, published in the Advances in Neural Information Processing Systems in 2023. Deep Patch Visual SLAM is by Lipson, Teed and Deng, published at the European Conference on Computer Vision in 2024. The first is odometry, the second adds the map.

The method itself is a learned one. The name says what it is, patch based visual odometry, and the approach is to match dense patches between frames rather than sparse keypoints. The repository layout shows the pieces: a dpvo directory with the model and the fast bundle adjustment, an altcorr subdirectory for the correlation, a fastba subdirectory for the bundle adjustment, and a lietorch directory for the LieTorch backends, which is where the learning based optimisation lives.

The build script confirms that this is a hybrid rather than a pure Python project, because the setup file builds three CUDA extensions: the correlation extension from a C++ and a CUDA source, the bundle adjustment extension from three sources, and the LieTorch backends from a CPU, a CUDA and a C++ source, all with the optimisation level set to three and all including the vendored Eigen headers.

There is also a Colab notebook linked from the README and a separate Docker image repository, which are the two fastest ways to see the thing work before you build anything.

The setup: conda, Eigen, CUDA extensions and two gigabytes

The installation is four steps and one large download.

The code was tested on Ubuntu 20 and 22 with CUDA 11 and 12, which is the first thing to check, because the extensions are compiled with the toolkit on your machine.

Clone the repository recursively, since the submodules carry the third party code, and then create the conda environment from the environment file and activate it.

git clone https://github.com/princeton-vl/DPVO.git --recursive
cd DPVO

conda env create -f environment.yml
conda activate dpvo

Eigen is not taken from the system, it is downloaded and unpacked into the third party directory, and then the package is installed with pip.

wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty

# install DPVO
pip install .

Then the models and data, which the README puts at about two gigabytes, fetched by a script in the repository.

./download_models_and_data.sh

That script is worth knowing about on its own, because the pretrained models are also published separately on Google Drive, so if you only want to run the demo you can fetch the model archive instead of the full data set.

The recommended viewer is Pangolin, and the README is specific that you need the custom version included with the repository rather than a system install, with a prerequisite script, a cmake configure, a parallel make and a system install, and then the viewer itself installed with pip. The Docker image is offered as the escape hatch for installation issues.

One demo script, and the output formats it can write

The demo section is where the project is most usable, because the interface is one Python script with a set of flags.

python demo.py \
    --imagedir=<path to image directory or video> \
    --calib=<path to calibration file> \
    --viz # enable visualization
    --plot # save trajectory plot
    --save_ply # save point cloud as a .ply file
    --save_trajectory # save the predicted trajectory as .txt in TUM format
    --save_colmap # save point cloud + trajectory in the standard COLMAP text format

The output flags are the interesting part, because they are the difference between a demo and a tool you can compare against other work. The trajectory can be written in the TUM format, and the point cloud and trajectory can be written in the standard COLMAP text format, which means the output can be loaded into COLMAP and inspected in the same tool other papers use.

The README notes that you can save the completed reconstructions and view them in COLMAP, and that the viewer is needed to visualise the reconstructions in real time.

Three dataset specific examples follow, and the difference between them is the stride. The iPhone example uses a movie file with a stride of five, the TartanAir example uses a stride of one, and the EuRoC example uses a stride of two. The stride is how many frames are skipped, so it is the knob that trades accuracy against speed, and the fact that a phone video needs a much larger stride than a rendered dataset says something about the difficulty of the input.

The SLAM backends, and the optional classical path

Running DPVO with a SLAM backend, which the README calls DPV-SLAM, is a single option added to any of the evaluate scripts or to the demo.

--opts LOOP_CLOSURE True

And there is a second, classical backend that can be enabled if it is installed.

--opts CLASSIC_LOOP_CLOSURE True

The classical backend is described as provided for closing very large loops, and it requires extra installation in three steps. The OpenCV C++ API comes from the system packages on Ubuntu, the DBoW2 bag of words library is built with cmake and installed, and the image retrieval component is installed from the repository with pip.

sudo apt-get install -y libopencv-dev

cd DBoW2
mkdir -p build && cd build
cmake .. # tested with cmake 3.22.1 and gcc/cc 11.4.0 on Ubuntu
make # tested with GNU Make 4.3
sudo make install
cd ../..

pip install ./DPRetrieval

That optional path is a useful thing to know about, because it is the traditional approach to the problem the learned tracker struggles with. A learned patch matcher tracks well over short sequences, and a bag of words approach with image retrieval is what closes a loop after the tracker has drifted far enough that direct matching fails. The classical backend is the older machinery, and it is kept as an option rather than a requirement.

Evaluation and training: the protocol and the command

The evaluation section is a set of scripts, one per dataset, and they all take the same shape.

There are scripts for TartanAir, EuRoC, TUM-RGBD, ICL-NUIM and KITTI, each run with a number of trials and the same output flags, for example the TartanAir one runs on the validation split with a plot and a saved trajectory, and the others take the trial count and the same two flags.

python evaluate_tartan.py --trials=5 --split=validation --plot --save_trajectory

python evaluate_euroc.py --trials=5 --plot --save_trajectory

The repeated trials matter for the interpretation of the numbers, since running a stochastic method five times and reporting the result is a protocol rather than a single run. Up to date result logs for these datasets are kept in the logs directory in the repository, which means the reference numbers are available for comparison.

Training is a single command, and it assumes the download script has been run so that the datasets are in place.

python train.py --steps=240000 --lr=0.00008 --name=<your name>

The README explains that log files are written to a directory under your name, and that the model is run on the validation split every ten thousand iterations. The expected directory layout is shown, with a datasets directory containing the TartanAir pickle and the extracted TartanAir sequences.

So the whole research loop is present in the repository: the training command, the evaluation scripts, the reference logs and the weights.

The state of the project, and what it costs to run

A few things to weigh before you plan around this.

The change log is short and dated, with an initial release in August 2022, a link to the Docker image in September 2022, the Colab notebook together with the TUM and ICL-NUIM evaluation code and the output saving flags in March 2023, and DPV-SLAM with the updated output saving utilities in July 2024. The last push to the repository was on 2026-09-20, and there are no tagged releases, so the repository is the release.

The cost is real. There is a conda environment, a vendored Eigen, three CUDA extensions to compile, a viewer to build or a Docker image to pull, and roughly two gigabytes of models and data. That is a heavier setup than a Python package, and it is the price of a project whose core is a compiled learned matcher.

The alternative to using this code at all is to work from the published papers, and for evaluating the method the repository is the more useful of the two, because the evaluation scripts and the reference logs let you check a number rather than trust it. For a working system, the classical path in the repository, with DBoW2 and image retrieval for loop closure, is a reminder that a purely learned front end still needs something to close large loops.

The acknowledgement is worth a line too, since it says where the viewer came from. The viewer is adapted from DSO, so the visualisation layer is inherited rather than written, and that is a good sign for the effort involved in getting the reconstructions in front of a person.

Editorial conclusion

Adopt DPVO if you are doing research on visual odometry or SLAM and want a learned dense patch matcher with a published evaluation protocol you can rerun, because the repository ships the training code, the evaluation scripts and the trained weights. Do not adopt it as a library inside a product, since the code is a research release with a conda environment, a viewer and a two gigabyte model download. Before you start, check your CUDA version against the tested range of 11 and 12, and budget the setup time, because the extensions compile against a vendored Eigen and the classical backend pulls in three more components.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. princeton-vl/DPVO on GitHub
4. README