A three-stage pipeline from everyday hand videos to robot fingers

Where dexterous manipulation data has to come from

This is the code release for a research paper on obtaining dexterous manipulation data from ordinary videos of people handling objects. It is organised as three stages, each in its own directory, and the third one ends with commands being streamed to physical robot arms.

The problem it addresses is the bottleneck in the field. Training a robot hand to manipulate objects needs demonstrations, and demonstrations have historically come from two expensive places. A teleoperation rig puts a human operator behind gloves or controllers and records what the robot does, which produces data in exactly the right format and produces it slowly, one operator-hour at a time. Motion capture puts markers on hands in an instrumented room, which is faster but constrains what can be captured to whatever fits in the room.

Video of people using their hands, by contrast, exists in effectively unlimited quantity and costs nothing to collect. What it lacks is everything that makes the other two useful: no joint angles, no object pose, no contact information, no depth, and no guarantee the hand is even fully visible.

The pipeline here is an argument that those missing quantities can be recovered from the video instead of instrumented at capture time. Whether that recovery is accurate enough is the paper's question; the repository is what the authors used to ask it.

Three directories that form a strict chain

The README describes each part of the pipeline as contained in its own folder, and the three are ordered rather than independent.

Reconstruction takes a video of a hand manipulating an object and produces object and hand reconstruction with six-degree-of-freedom pose tracking. Retargeting maps that reconstructed demonstration onto a robot hand, and the README states it consumes the reconstruction output directly. Deployment replays a retargeted demonstration on real hardware, turning the retargeting output into a joint trajectory for the arms.

That directory structure is the most useful thing about the release for anyone trying to reproduce or extend it. Research pipelines of this shape are frequently published as one entangled repository where the stages share a configuration file, a data format and a set of global assumptions, and where a failure at the end tells you nothing about which step went wrong.

Separating them means the intermediate representations have to be real artefacts written to disk rather than objects passed in memory. That in turn means a user can run reconstruction alone and inspect the result, or bring their own reconstruction to the retargeting stage, or take a retargeted trajectory to hardware that is not the authors'.

Each directory carries its own README with full details, and the top-level document is deliberately a table of contents rather than a manual.

An assembly of published models, not a new one

The reconstruction stage is described as a chain of named components in sequence: segmentation, mesh generation, point map estimation, hand reconstruction, point tracking, a guided diffusion step for tracking, and optionally a projection step.

That is seven separately developed models composed into one pass, and recognising it as composition rather than as a single network is the right way to read this repository. The contribution is not a new perception model. It is the demonstration that existing models, arranged in the right order with the right glue, recover enough structure from an uninstrumented video to drive a robot hand.

The ordering has an internal logic worth tracing. Segmentation establishes what in the frame is the object. Mesh generation gives that object a three-dimensional form. Point maps supply the scene geometry the object sits in. Hand reconstruction recovers the hand separately, because hands and objects occlude each other constantly and a method that treats them jointly from the start inherits both problems. Point tracking then provides correspondence across frames, and the diffusion step guides the tracking rather than replacing it.

The word guided is doing real work there. Using a generative model to constrain a tracking problem, instead of asking it to produce the answer, is the conservative choice, and it is the one that keeps the output tied to the observed video.

Physics, and why retargeting is more than a mapping

The retargeting stage is listed as dataset processing, convex decomposition, scene generation in the physics engine's own format, inverse kinematics, and sampling-based model predictive control in a physics simulator.

Two of those steps explain why this stage is harder than it sounds. Convex decomposition appears because a mesh reconstructed from video is an arbitrary surface, while physics engines compute contact against convex shapes, so the reconstructed object has to be broken into convex pieces before it can be touched in simulation at all.

The presence of model predictive control after inverse kinematics is the more telling detail. Inverse kinematics answers where the joints must be for the hand to reach a given pose, which is sufficient for a robot moving through free space. It is not sufficient for manipulation, where the object moves in response to contact, small errors change what is being touched, and a sequence of individually correct poses can add up to dropping the object. A sampling-based controller running against a simulated version of the scene searches for actions that keep the task working, which is a different and more expensive question than solving for a pose.

A human hand and a robot hand also differ in proportions, joint count and range of motion, so the mapping between them is never exact. The retargeting stage is where that mismatch has to be resolved into something a specific hand can actually execute.

Submodules with the changes already made

One sentence in the README carries more practical weight than its length suggests: external code references are provided as git submodules with the authors' changes baked in.

Anyone who has tried to run a research pipeline that composes seven upstream projects knows the failure this prevents. Each dependency has its own version history, its own environment requirements and its own incompatible expectations about data formats, and each one keeps moving after the paper is published. A repository that lists dependencies by name and asks the user to install them is a repository that stops working within months, and the user is left guessing which of seven projects changed.

Pinning submodules with modifications already applied means the version that produced the paper's results is the version a reader gets. It is the honest choice for a code release whose purpose is reproducing published work.

The cost is real and worth stating. Those forks drift from their upstreams, so improvements made later by the original authors do not arrive, and a user who wants a newer version of any one component is on their own reconciling it. For a research artefact, reproducibility is the right side of that trade, but it does mean this is a snapshot rather than a maintained stack.

The repository carries a submodules configuration file at its root, consistent with that description, and publishes no releases, so the default branch is the reference point.

The hardware at the end of the pipeline

The deployment stage is the one to read carefully before planning around this repository, and the README is commendably plain about it.

It describes a barebones replay pass that converts a retargeting output into a joint trajectory for a specific dual-arm setup, streamed to named arms fitted with named robotic hands. Barebones is the authors' own word, and the specificity of the hardware is the constraint.

That makes the three stages differently portable. Reconstruction works on video and should carry anywhere. Retargeting targets a robot hand and would need work for a different one, but the stage exists precisely to absorb that kind of difference. Deployment is written for the machines in one laboratory, and a reader with other arms should expect to write their own equivalent rather than to configure this one.

That is a normal and defensible boundary for a paper release. It is worth naming because the presence of a deployment directory can suggest a turnkey path to hardware that does not exist here for anyone whose bench does not match.

The repository is MIT licensed, which is unusually permissive for robotics research code and allows commercial use of what the authors wrote. That licence does not extend to the submodules, which carry whatever terms their own projects set, so anyone building on this commercially has several licences to check rather than one. This is not legal advice.

The project reports 409 stars, 41 forks and a single open issue, with the last push on 2026-08-02. A project page and an arXiv paper are linked from the README, and reading the paper first will make the directory structure considerably easier to follow.

Editorial conclusion

This release is aimed at manipulation researchers who want to reproduce the paper or reuse one of its stages, and the directory separation makes the second of those genuinely practical, since reconstruction outputs are artefacts a reader can inspect or replace rather than objects passed inside one process. Expect to read the paper and the three per-directory documents rather than the short top-level README, which is a table of contents by design. Treat the pinned submodules as the point rather than as neglect, because they are what keeps a seven-model pipeline runnable after publication, and plan to write your own deployment stage unless your bench happens to hold the same dual-arm setup the authors used.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. malik-group/do-as-i-do on GitHub
4. Project website
5. README