DeepLabCut: markerless pose estimation when you cannot define the keypoints in advance

The problem is keypoint definition, not pose estimation

Human pose estimators ship with a fixed skeleton: shoulders, elbows, wrists, ankles. The moment your subject is a cheetah, a fly, or a mouse whose paw you want to track at a specific joint, that skeleton is the wrong answer. DeepLabCut takes the opposite approach. You decide what to label, the README states the toolbox is animal and object agnostic, and the network learns to find those specific points. The labelling burden moves onto you, but so does the definition of the problem. This is aimed at behavioural neuroscientists, ethologists and anyone else who needs trajectories for user-defined body parts rather than a generic skeleton. The repository topics list behavior-analysis, keypoint-tracking and labeling-tool alongside pose-estimation, which is an accurate summary of what the package contains: not just a model, but a project workflow around one.

What actually happens between labelled frames and tracked video

The pipeline is a supervised detector plus a tracking layer. You extract frames from your videos, open the labelling GUI, and mark the points you care about on each frame. Those annotations become the training set for a feature detector, which is the term the repository topics use for the network. Training produces a model that predicts, for any frame, the coordinates of your points. Inference then runs that model over the full video, and the tracking stage links per-frame detections into consistent trajectories across time, which is what turns a stack of coordinates into a usable time series. The important architectural consequence is that nothing about the model is species-specific. The same code path handles the mouse, cheetah and fly examples the README displays. What changes between projects is the annotation set and the network configuration, not the toolbox. The second consequence is that your annotation quality sets the ceiling on everything downstream. There is no self-supervised escape hatch in the material provided: the documentation describes a labelling step, and the labelling step is where your project's accuracy is decided.

Installing the PyTorch engine and the extras that matter

The README requires Python 3.10 or newer and gives a two-step install for the PyTorch engine. First install PyTorch itself, either through pip or through conda with a CUDA build, for example: conda install pytorch cudatoolkit=11.3 -c pytorch. Then install the toolbox with the GUI included: pip install --pre "deeplabcut[gui]". The headless variant is pip install --pre "deeplabcut", which drops the GUI and is the one you want on a training server with no display. The extras are where the real decision lives. The TensorFlow engine needs pip install "deeplabcut[gui,tf]" or the headless pip install "deeplabcut[tf]", and the README also offers CUDA-specific extras in the form of deeplabcut[tf-cu11] and deeplabcut[tf-cu12]. Those narrow extras exist because TensorFlow and CUDA version matching is the usual source of installation failures, and picking the wrong one produces an environment that imports but cannot see the GPU. Note the --pre flag on the documented commands: it installs pre-release builds, which is worth understanding before you pin a version in a shared environment.

The TensorFlow backend is on a deprecation path

The README states plainly that the project aims to deprecate the TensorFlow backend in version 3.2, with the release date to be determined. That is the single most important planning fact in the material. If you are starting a new project, the PyTorch path is the one with a stated future. If you are maintaining an existing TensorFlow-based pipeline, you are working against a clock whose deadline has not been set, which is a worse position than a scheduled one because you cannot plan the migration against a date. The release history supports the direction of travel: v3.0.0 landed in May 2026, v3.0.1 in July 2026, and the repository's last push is dated September 2026, so 3.x is the active line. Anyone with TensorFlow checkpoints and a custom training script should treat the engine switch as a real migration project rather than a flag change, and should verify current guidance in the installation docs rather than assuming the extras listed above will remain the recommended route.

Where the toolbox is the wrong tool

The failure mode is annotation cost, and it is not a small one. Because the model is trained on your labels, a project with a new body plan and no close match in the Model Zoo starts from a blank labelling task. The README points to a Model Zoo as a place to find pretrained networks, and the sensible workflow is to check there first: a network trained on a similar animal can reduce the number of frames you need to annotate. If nothing close exists, the labelling work is the project, and that cost scales with the number of keypoints, the number of frames and how much your subjects vary in appearance and pose. The second case is a team that needs a general human skeleton and nothing custom. A fixed-skeleton estimator gives you a working model with no labelling at all, and DeepLabCut's flexibility buys you nothing in exchange for the annotation effort. The third case is deployment. The README describes a Python toolbox with a GUI, conda files and pip extras, plus an optional TensorFlow install matrix. That is a research and analysis environment, not a lightweight edge runtime, and the material gives no indication that it is intended as one.

How it differs from a fixed-skeleton pose estimator

The difference is in what is fixed and what is learned. A general-purpose human pose estimator fixes the keypoint set and learns a mapping from images to that set, so you get results immediately and you get exactly those joints. DeepLabCut fixes the learning machinery and lets you define the keypoints, so you get trajectories for whatever you can see and label, and you pay for that with an annotation stage. The trade is not subtle: one approach gives you a model with zero setup and no flexibility, the other gives you full flexibility and a labelling task that sits on the critical path before any result exists. There is also a tracking dimension. Because DeepLabCut's output is per-frame detections linked into trajectories, the quality of the tracking depends both on detector accuracy and on how well the linking handles occlusion and identity swaps, which is a separate problem from getting good per-frame coordinates. A fixed-skeleton estimator applied frame by frame does not give you that trajectory layer for free either, but the failure modes differ, and the DeepLabCut documentation is the place to check how tracking behaves in your recording conditions.

Licence and maintenance cost

DeepLabCut is licensed under LGPL-3.0. The practical implication of that licence family is that if you modify the library itself and distribute your modified version, the modified library stays under LGPL-3.0. Using it as a dependency in your own research code is a different situation from redistributing a modified copy of it, and the boundary between the two is exactly the kind of question to put to whoever handles licensing at your institution rather than to a review article. This is not legal advice. On maintenance: the release cadence shown here is roughly one minor release every few months across 2026, with release candidates appearing before stable tags, and the documented install commands carry a --pre flag. That combination means version pinning matters more than usual. A conda environment file that pins the exact DeepLabCut version and the matching engine extra is cheaper to maintain than one that floats, because the TensorFlow deprecation in 3.2 will otherwise arrive as a surprise in a working environment. The README also points to an Image.sc forum tag and a Gitter channel for support, which is where version-specific installation problems are usually resolved.

Editorial conclusion

Adopt DeepLabCut when the features you need to track cannot be expressed by a pretrained human-pose model and you can afford to label frames yourself. Do not adopt it if you need a zero-annotation pipeline, or if you cannot ship LGPL-3.0 code alongside your own. Before committing, verify three things against the docs for the exact version you install: which engine you are on, since the README states the TensorFlow backend is targeted for deprecation in 3.2; whether your CUDA version is covered by a matching extra such as deeplabcut[tf-cu11] or deeplabcut[tf-cu12]; and whether the Model Zoo carries a network for a body plan close to yours, because that decides how many frames you will have to label before training is worth starting.

Official sources

1. DeepLabCut/DeepLabCut on GitHub
2. License: LGPL-3.0
3. Project website
4. README
5. Releases