MocapNET: monocular video to BVH in real time, and what it costs you

The gap MocapNET fills: BVH out, not joint coordinates

Most 2D-to-3D pose work ends with an array of joint positions in some camera-relative space. That array is not what an animator or a game engine consumes. Pipelines that drive a rig need a skeleton with named bones, a rest pose and rotation channels, which is what the Bio Vision Hierarchy format encodes. MocapNET's stated contribution is that it estimates the 3D pose directly in BVH, given 2D body joints estimated from monocular color images. The README describes two supporting pieces: a compact 2D pose NSRM representation, and a human body orientation classifier backed by an ensemble of orientation-tuned networks.

The audience is narrow and specific. It is researchers and technical artists who already have a 2D joint detector they trust, and who need the 3D lifting plus skeleton export step. It is also people prototyping low-cost motion capture, which is the phrasing that shows up in the searches around this project. If you want a library that takes a video file and hands back a rigged character with no intermediate representation, this is not that. The BVH file is the deliverable, and everything downstream of it (retargeting into Blender, into a game engine, into a different skeleton) is your problem and your tooling.

How the 2D joints become a BVH skeleton

The data flow has three visible stages. A 2D joint estimator produces body keypoints from a color image or a webcam frame. MocapNET's networks consume those keypoints in the NSRM representation and predict 3D pose, with a separate orientation classifier deciding which way the body faces, since a monocular view is ambiguous on that point and an ensemble of orientation-tuned networks handles the cases where a single regressor would flip. The output is written as BVH.

The repository layout reflects this split. There is a src directory, a dependencies directory, a dataset directory and a docker directory, plus initialize.sh, update.sh and revert.sh at the top level. The presence of revert.sh alongside update.sh is worth noting: it suggests the maintainers expect the update path to sometimes need undoing, which is a reasonable design for a research repository where dependency versions move. The CMakeLists.txt at the root is the entry point for the C++ build.

Version 4 is a different codebase. The README states that the whole v4 codebase was written from scratch in Python and lives in its own mnet4 branch, including 3D rendering through Blender python scripts. That means the C++ code you see on the default branch and the Python code on mnet4 are not two views of one system. They are two systems sharing a name and a research lineage.

Installing MocapNET and getting a first BVH file

There are two routes in the README. The fastest is the Google Colab notebook on the mnet4 branch, which the README calls a one click deployment and which is linked with a Colab badge. If you only want to see the method produce output, start there, because it avoids the dependency chain entirely.

The second route is a local build. The repository root contains CMakeLists.txt and an initialize.sh script, which is the conventional order for this project family: fetch and prepare dependencies first, then configure the build. The README does not spell out the exact arguments for either script, so read the script contents before running them rather than assuming flags.

./initialize.sh
mkdir build
cd build
cmake ..
make

After the build completes you should have the binaries under the build tree. The dependencies directory is where the project expects third-party code to land, which is why initialize.sh runs first.

For the Python v4 path, the branch matters more than the command. The README points to a Mediapipe plus MocapNET v4.0 script at src/python/mnet4/mediapipeHolisticWebcamMocapNET.py for live webcam capture.

git clone https://github.com/FORTH-ModelBasedTracker/MocapNET.git
git checkout mnet4
python3 src/python/mnet4/mediapipeHolisticWebcamMocapNET.py

That script is described as handling body, hands and face together. The README reports a sustained 30Hz on a Lenovo Ideapad Pro 5 14IMH9; treat that as the author's machine and setup, not a guarantee for yours, since nothing in the README describes the resolution, the lighting or the CPU and GPU configuration behind that number. A docker directory exists in the repository for containerized builds, but the README does not document the container workflow, so you would be reading the Dockerfiles directly.

Where MocapNET breaks or is the wrong choice

The monocular constraint is the root of most failure modes. A single camera cannot resolve depth reliably, and self-occlusion, loose clothing and fast limb motion all degrade the 2D joints that everything downstream depends on. MocapNET inherits those errors and then adds a skeleton-fitting step on top, so a bad 2D estimate becomes a bad BVH file with plausible-looking bone rotations. Nothing in the README describes a confidence score or a rejection path for frames the network is unsure about.

The orientation classifier and its ensemble exist precisely because facing direction is ambiguous from one view. That is a mitigation, not a solution. Expect the body to occasionally snap to the wrong facing direction on ambiguous frames, and expect that error to persist across a short window rather than self-correct instantly.

There is also a licensing ambiguity. The repository metadata reports the license as NOASSERTION, which means the hosting platform could not map the license file to a known identifier. The repository does contain a license.txt, and reading it is the only way to know your terms. For a research project that has been through academic publication and EU funding programs, that is common, but it is a real step you cannot skip if you plan to ship anything.

The last push to the default branch was on 2026-08-13, so the repository is not dormant. The README's news entries, however, show a long stretch where the author states that repository maintenance was subpar while finishing a PhD thesis, and that v4 lives on a separate branch rather than being merged. Branch fragmentation is a maintenance cost you inherit: fixes on one branch do not automatically reach the other.

MocapNET against Pose2Sim and Mediapipe

Pose2Sim is the closest thing to a direct alternative in the searches around this project, and the difference is architectural rather than incremental. Pose2Sim is built around multi-camera markerless capture: it triangulates 2D keypoints from several synchronized views before fitting a model. MocapNET is defined for monocular color images. If you have two or more cameras and can calibrate them, the multi-view route removes the depth ambiguity that MocapNET has to guess at, and the resulting 3D joints are constrained by geometry rather than by a learned prior. If you have one webcam and no calibration budget, that route is closed to you and MocapNET's learned lifting is the available option.

Mediapipe is a different relationship: it is not a competitor on the output side, it is an input source. The README's own news entry notes that the 2D joint estimator offered with MocapNET 3 does not contain hands, and that a transition to Mediapipe Holistic is needed for a better live webcam demo. That is the project telling you its own 2D front end is the weak link. You can take Mediapipe's keypoints and feed them into MocapNET's lifting and BVH export, which is exactly what the mnet4 script does. The practical comparison is therefore not MocapNET versus Mediapipe, it is whether you want MocapNET's BVH export on top of Mediapipe's joints, or Mediapipe's own world landmarks with your own retargeting code.

Maintenance, licensing and what an upgrade actually involves

The default branch received a push on 2026-08-13. Version 4 has its own branch, mnet4, and the README describes it as a from-scratch Python rewrite rather than an incremental change to the C++ code. That has a direct cost consequence: moving from v3-era code to v4 is not a dependency bump, it is a re-integration against a different language, different entry points and a different rendering path through Blender scripts.

The update mechanism is unusual and worth flagging. The repository ships update.sh and revert.sh at the top level alongside initialize.sh. An update script paired with a revert script implies in-place modification of the working tree, most likely of the dependencies directory, rather than a versioned package dependency. If you vendor this repository into a larger build, run the scripts in a copy or a container first, because an in-place update that touches dependencies is the kind of thing that silently breaks a build you thought was pinned.

On licensing, the platform reports NOASSERTION. That is not a license, it is the absence of a recognized one. Read license.txt in the repository root before you build anything into a commercial pipeline. I am not giving legal advice here, only pointing at the file that determines the answer.

Editorial conclusion

Adopt MocapNET if your output target is a BVH skeleton and you are willing to build from source or use the Colab notebook, because most pose estimators stop at joint coordinates and leave the skeleton retargeting to you. Do not adopt it if you need a supported product with release notes and a deprecation policy, or if your capture setup is multi-camera, since the method is defined for monocular color images. Before committing, verify three things: which branch you are on, whether the 2D joints come from the bundled estimator or from Mediapipe Holistic, and whether the license.txt file in the repository root matches the terms you need, because the repository metadata does not resolve to a standard SPDX identifier.

Frequently asked questions

Official sources

1. FORTH-ModelBasedTracker/MocapNET on GitHub
2. Issues
3. Project website
4. README