SiLK learns keypoints without labels and ships with a trained checkpoint

A keypoint detector that needs no annotations

SiLK, which stands for Simple Learned Keypoint, is a self-supervised framework for learning keypoints, and the authors are explicit about the priorities. The project focuses on simplicity and flexibility, while also providing state-of-the-art and competitive results on existing benchmarks. The distinction matters. A supervised keypoint detector needs a dataset of points marked by hand, which for a domain like remote imagery, medical scans or industrial parts means annotating a domain-specific set before you can train anything. A self-supervised model learns its points from the images themselves, so the same weights transfer to a task nobody annotated. That is why the model is described as a foundation backbone rather than a point detector for one benchmark, and why the evaluation in the README spans image matching benchmarks and, further down, scene reconstruction. The paper is the ICCV 2023 conference version, and the repository is the official implementation with the authors Pierre Gleize, Weiyao Wang and Matt Feiszli.

The training setup is pinned to 2021 packages

The requirements file is the first thing to read and it is a period piece. It pins torch at 1.11.0+cu113 and torchvision at 0.12.0+cu113, jax and jaxlib at 0.3.25 with the cuda112 extra, scikit-image at 0.19.3, pytorch-lightning at 1.5.2, albumentations at 1.3.0, hydra-core at 1.2.0 with omegaconf at 2.2.3, einops at 0.4.1, kornia at 0.6.6 and torchmetrics at 0.6.0.

torch==1.11.0+cu113
torchvision==0.12.0+cu113
jaxlib[cuda112]==0.3.25
jax==0.3.25
hydra-core==1.2.0
omegaconf==2.2.3
pytorch-lightning==1.5.2

On top of that sit the development utilities loguru, autoflake, black at 21.10b0, matplotlib, pdoc3, docopt, schema, pycocotools, pytorchvideo, imageio and diskcache. Two of these pin the environment hard: hydra-core and omegaconf are the configuration system for the training runs, and the pinned versions are old enough that a fresh environment will need the matching PyTorch build index. The Makefile shows how the project expects that to be handled, with the dev_install target pointing pip at the cu113 torch wheel index and the jax cuda release storage, and a comment in the command explaining that the extra find-links are required to locate the pytorch and jax packages.

Two GPUs and five hours, and a conda environment

The stated requirements are short and concrete. Conda should be installed in order to set up the silk environment, and two GPUs are required to train SiLK. The released code has been tested on Linux with two Tesla V100-SXM2 GPUs and takes about five hours to train. The Makefile has the environment targets to match, with conda_env_init creating a named silk environment at python 3.8.12, conda_export writing the environment to envs/dev.yml, and conda_update applying that file with the prune flag. There is also a doc target that runs pdoc over the silk package to generate the HTML documentation into the doc/ directory, with the dependency on the shell find over the Python sources so the docs rebuild when the code changes. The practical read is that this is a research codebase with a reproducible environment definition rather than a library you install into an existing project, and the packaging is minimal, since the setup.py names the distribution SiLK Keypoint Library at version 1.0 with the single package silk and takes its install requirements from that same requirements.txt.

Pretrained checkpoints live outside the repository

Pretrained models are provided, and the results section says exactly which ones the published numbers use. The VGG-4 backbone checkpoint, named pvgg-4.ckpt, was used for the HPatches and IMC tables, while the coco-rgb-aug checkpoint was used for the ScanNet tables. Both are hosted on Facebook's public file storage rather than in the repository, so a fresh clone does not give you the weights and you need the links to work to reproduce anything. For the Image Matching Challenge 2022 results the project goes further and provides a Kaggle notebook, which is the piece to look at if you want to see the full evaluation path rather than just the number. The results themselves are presented as images of tables, and the README is careful to say that the numbers were computed with those specific checkpoints, which is a level of care that is rarer than it should be in research repositories and makes the tables usable as a baseline rather than as marketing.

The usage docs are a path, not a manual

The documentation is organised as a set of task pages, and the README is honest about their limits, saying the documentation is non-exhaustive and asking people to open an issue if clarification is required, with the promise that documentation will be added if the community requires it. The pages are setup for the python environment, dataset setup, training, adding a backbone, running the evaluation pipeline, running inference, converting SiLK to TorchScript, importing SiLK features to COLMAP, and a FAQ. Two of those are the ones that make this a pipeline component rather than a research artifact. The TorchScript conversion is what lets a trained model leave the Python training stack and run somewhere faster. The COLMAP import path is what lets keypoints become tracks in a structure-from-motion pipeline, which is the natural consumer of a keypoint detector in the 3D vision world. The rest of the repository layout backs up that reading, with lib/, silk/, bin/, scripts/, notebooks/, external/, etc/, var/ and a gitmodules file for the external dependencies.

Where SiLK is the wrong tool

Three cases where this is not the answer. The first is a single GPU or a CPU-only machine, because two GPUs are required to train and the released code was tested on two Tesla V100-SXM2 GPUs. Pretrained weights remove the training cost but not the environment cost, and the environment is the older one. The second is a modern PyTorch stack. Anyone on torch 2.x has to build a separate environment around torch 1.11 with the cu113 wheel index, and the pinned hydra, pytorch-lightning and jax versions compound that. A research group that wants to compare SiLK against a modern baseline will spend more time on dependency resolution than on the comparison. The third is a task where you need dense, sub-pixel accurate correspondences rather than sparse interest points, since a keypoint model is the wrong abstraction for dense matching and the repository does not offer a dense alternative. There is also a licensing dimension to weigh. SiLK is licensed under the GNU General Public License version 3, which is a heavier obligation than the permissive licences common in computer vision research, and for a model you intend to ship inside a product that is a decision to make early rather than at release time.

Editorial conclusion

Adopt SiLK if you are building an image matching or local feature pipeline and want a keypoint detector that needs no human annotations, along with pretrained weights and the surrounding tooling to evaluate, export and feed it into COLMAP. Skip it if your environment is on a modern PyTorch, because the requirements pin torch 1.11.0 and jax 0.3.25, and expect two GPUs and about five hours of training for a run from scratch. Before you start, read doc/usage/setup.md, decide whether you want the pinned VGG-4 backbone configuration the published results use, and check that the pretrained checkpoints still resolve, since the model files are hosted outside the repository.

Frequently asked questions

Official sources

1. facebookresearch/silk on GitHub
2. Issues
3. License: GPL-3.0
4. README