TapNet: Google DeepMind's Point Tracking Repository, Model by Model
Tracking Any Point (TAP)
At a glance
- What is it?
- TapNet bundles several generations of Tracking Any Point models, the TAP-Vid and TAPVid-3D benchmarks, and the RoboTAP robotics extension in one Apache-2.0 repository. The hard part is not running a demo, it is picking which of the seven linked models matches your latency, occlusion and frame-length requirements.
- Who is it for?
- Adopt TapNet if you need a reference implementation of point tracking plus the TAP-Vid or TAPVid-3D evaluation metrics, and you are willing to choose between the offline TAPIR path, the causal online TAPIR path and the TAPNext family yourself. Do not adopt it expecting a packaged inference service or a single recommended checkpoint, because the README presents seven model pages with different training procedures and no decision table.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 55 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem TapNet addresses: tracking a chosen pixel, not a detected box
Object detection and segmentation answer where a thing is. TapNet answers where one specific point on that thing went, frame after frame, including through deformation and partial occlusion. The README frames this as Tracking Any Point, and the repository's own description of TAP-Vid calls it a benchmark for models that perform this task, with ground-truth points for both real and synthetic videos. That distinction matters because a bounding box can survive a change in appearance while an individual point on a rotating surface cannot be recovered by appearance matching alone. The intended audience is visible in the topic list and the linked projects: computer-vision researchers who need a benchmark to report against, and robotics engineers, since RoboTAP uses TAPIR point tracks to execute manipulation tasks through imitation in the real world. If your task is tracking a whole object or a semantic region, this is a heavier tool than you need.
Two-stage matching and refinement, and how TAPNext departs from it
The README describes TAPIR as a two-stage algorithm. Stage one is a matching stage that independently locates a candidate point match for the query point on every other frame. Stage two is a refinement stage that updates both the trajectory and the query features based on local correlations. The word independently is the architectural point: candidate matches are not chained frame to frame in the first pass, so an error on frame 40 does not automatically propagate into the candidate set for frame 41. The refinement pass then reconciles those candidates into a trajectory using local correlation. TAPNext abandons that structure. The README states it formulates the TAP problem as next token prediction and tracks points simply by propagating information through a network. That is a different data flow, per-frame and recurrent rather than all-frames-at-once. TAPNext++ is described as an improved TAPNext checkpoint with 40x longer stable tracking performance that allows tracking through occlusions and shows strong re-detection capabilities, fine-tuned on 1024-frame synthetic sequences. Note that these are the repository's characterisations of the models, not independent measurements.
Choosing between the offline, online and per-frame checkpoints
The demo list is the real decision surface, and it splits three ways. Standard TAPIR, in colabs/tapir_demo.ipynb, is described as the most powerful TAPIR / BootsTAPIR model that runs on a whole video at once, and the README says results for this model are mainly what the paper reports. Online TAPIR, in colabs/causal_tapir_demo.ipynb, is the sequential causal variant that allows online tracking and, per the README, can run in real time on a GPU platform. BootsTAPNext, in colabs/tapnext_demo.ipynb, is described as the most powerful TAPNext model that runs online per-frame. Then TAPNext++ in colabs/torch_tapnextpp_demo.ipynb adds occlusion tracking and re-detection. The trade-off is explicit in the wording: the strongest reported model needs the whole video, and the models that run causally are different checkpoints. If your pipeline is a camera feed, you are not using the model whose numbers appear in the paper. That gap is the single most important thing to understand before adopting TapNet, and the README does not quantify it.
Running TapNet: colab notebooks, checkpoints and the Jax versus PyTorch split
The lowest-friction path is the hosted colab notebooks, which accept an uploaded video for point tracking. The README also states you can clone the repo and run on your own hardware, including a real-time demo. Checkpoints are listed for TAP-Net, the baseline from the TAP-Vid paper, plus TAPIR and BootsTAPIR, in both Jax and PyTorch. That dual-format availability is stated for those three only. For TAPNext, the PyTorch notebook is described as a re-implementation that contains the exact architecture and weights as the Jax model, which is a claim worth verifying yourself rather than assuming across the whole family. Training instructions in the repository cover TAP-Net and TAPIR on Kubric, so the training path is narrower than the inference path. The primary language of the repository is Jupyter Notebook, which tells you where the working examples live: the colabs directory, not a Python package with a documented CLI. There is no install command, no pip package name and no configuration key list in the supplied README, so I cannot give you one.
Where the repository stops short of being an integration
TapNet is a research repository with demos attached, and several practical things are simply absent from the material. There is no documented Python API surface, no inference server, no batching guidance and no latency table for any checkpoint on any hardware. The real-time claim for online TAPIR is qualified by the README as running on a GPU platform, with no card named. The training instructions cover TAP-Net and TAPIR on Kubric, which means the newer TAPNext and TAPNext++ checkpoints have no training recipe here; you get weights and a fine-tuning description, not a reproducible pipeline. BootsTAP is described as a training procedure using large unlabeled real-world video with consistency across spatial transformations, corruptions and query point choices, applied to TAPIR to create BootsTAPIR, and the README notes the best TAPNext checkpoint was fine-tuned using that procedure. So the strongest models depend on a training stage whose data is not in this repository. If you need to retrain on your own domain, that is the wall you hit.
TRAJAN and the benchmarks are a separate offering from the trackers
It is easy to read the project list as one product. It is not. TRAJAN is described as a point trajectory autoencoder that conditions on a set of support point trajectories and reconstructs a held-out set of query points. Its embedding space is intended for comparing distributions of videos, comparing motion trajectories across videos independent of object appearances, and evaluating the realism and consistency of generative video model outputs. That is an analysis tool, not a tracker, and it has its own colab demo. The benchmarks are a third thing: TAP-Vid provides ground-truth points and evaluation metrics for the 2D task, and TAPVid-3D provides metrics plus sample evaluation code for 3D point tracking, with the README stating the benchmark contains over one million computed ground-truth trajectories on more than four thousand real-world videos. If you only need to score someone else's tracker, the metric code is the part of this repository you want, and you can ignore the checkpoints entirely.
Alternatives and the real difference in approach
The obvious alternative is optical flow, and the distinction is structural rather than a matter of accuracy. Classical dense flow estimates a vector field for every pixel between two consecutive frames. TapNet tracks a sparse set of user-chosen query points over an arbitrary frame range and reports visibility, and TAPIR's first stage evaluates candidate matches against every other frame rather than only the adjacent one. That is why long-range tracking and re-detection after occlusion are even expressible in this framework; a two-frame flow field has no notion of a point that disappeared and came back. The cost is that you must choose query points, and you get answers only about those points. Within the repository itself, the alternative to the offline TAPIR path is the causal online path, and that is a genuine fork in behaviour, not a performance setting. If your video is short and already recorded, the whole-video model is the one the paper reports. If it is a live stream, you are on the causal model and should treat the published numbers as belonging to a different configuration.
Licence, maintenance and what a version bump costs you
The repository is Apache-2.0, which permits commercial use and modification and includes a patent grant, but I am not a lawyer and this is not legal advice; check how the licence interacts with any pretrained weights you redistribute, since model weights are often governed separately from code. On maintenance, the repository is not archived and the last push recorded is 2026-07-22, so it is active. No releases were retrieved, which means there is no tagged version to pin against. That has a concrete consequence: cloning main gives you whatever the notebooks reference at that moment, and a checkpoint path or notebook cell can change without a version number to notice. If you build on this, record the commit hash you cloned alongside any results, because there is no release tag to fall back to. The upgrade cost is mostly re-validating which checkpoint you are actually loading, given that the model family has grown to seven linked projects with overlapping names and no release notes to diff between.
Editorial conclusion
Adopt TapNet if you need a reference implementation of point tracking plus the TAP-Vid or TAPVid-3D evaluation metrics, and you are willing to choose between the offline TAPIR path, the causal online TAPIR path and the TAPNext family yourself. Do not adopt it expecting a packaged inference service or a single recommended checkpoint, because the README presents seven model pages with different training procedures and no decision table. Before committing, open colabs/tapir_demo.ipynb and colabs/torch_tapnextpp_demo.ipynb side by side, confirm which one matches your frame budget (whole-video versus per-frame), and check the checkpoint section for whether the weights you need exist in both Jax and PyTorch, since only TAP-Net, TAPIR and BootsTAPIR are documented as shipping in both.
Community notes