LATENT releases the tennis motion tracking half of a humanoid pipeline

A tracking pipeline first, a tennis policy later

LATENT is the official implementation of the paper Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data, a collaboration between Tsinghua and Galbot. The project description calls it an open source humanoid robot learning pipeline covering motion tracker pre-training, online distillation and high-level policy learning, simulated in MuJoCo with multi-GPU parallel training support.

That is the ambition. What exists in the repository today is the first stage. The news entry dated March 13, 2026 records that the tracking codebase and a small subset of the human tennis motion data were released, and that the motions can now be tracked with the pipeline described in the paper.

The project list makes the boundary explicit. Two items are ticked: the motion tracking codebase and a small subset of the human tennis motion data. The unticked items are the ones that would make the paper's full claim reproducible: all the human tennis motion data used, pretrained trackers for all released data, the DAgger online distillation codebase, a pretrained latent action model trained on that data, the high-level tennis-playing policy training codebase, the sim2real designs for that policy, and more pretrained checkpoints.

The code is built on top of `jax`, `brax`, `loco-mujoco`, `mujoco_playground` and `OpenTrack`, and the tracking stage itself is based on OpenTrack. The paper is on arXiv at 2603.12686, and the project has its own website page.

Four environment variables and a uv sync to start

Setup is short and mostly delegated to `uv`. The repository is cloned, dependencies are synced, and a `.env` file supplies four variables: `GLI_PATH` pointing at the absolute project path, plus `WANDB_PROJECT`, `WANDB_ENTITY` and `WANDB_API_KEY` for experiment tracking.

The clone and the sync are the two commands that matter:

   git clone [email protected]:GalaxyGeneralRobotics/LATENT.git

   uv sync -i https://pypi.org/simple 

After that the retargeted tennis data has to be downloaded and placed under `storage/data/mocap/Tennis/`, with the folder layout holding per-player directories of `.npz` motion files. Assets are then initialised with a single script:

   python latent_mj/app/mj_playground_init.py

Running anything afterwards means activating the environment and loading the variables, so every session starts with the same two commands:

source .venv/bin/activate; source .env;

The dependency picture is heavy and pinned hard. The manifest requires Python 3.12.9 exactly, and pins numpy 2.1.3, jax 0.4.38 with the CUDA 12 variant, flax 0.10.4, brax 0.12.3, mujoco 3.3.1, tensorflow 2.19.0, torch 2.9.0, onnxruntime 1.23.2, and wandb for logging. There is also osqp for the inverse kinematics solver, and the project uses ruff with a line length of 120 and the py312 target.

The preprocess script overwrites the motion files it reads

Training on your own motion data is optional, and the documentation is direct about what it involves. You put `.npz` files under `storage/data/mocap/<your_dataset_name>/UnitreeG1` and run a preprocessing script that does two things: it aligns the frequency of the original motion data to the desired control frequency, 50Hz by default, and it recalculates velocities, angular, linear and joint, plus other state features, based on that aligned frequency.

The script overwrites the original motion files. That is the line to read twice, because it means the retargeted data you downloaded is modified in place the first time you run it, and the alignment is not reversible without a fresh copy.

For a single GPU the invocation is:

  python scripts/process_motion/preprocess_motion.py --task G1TrackingGeneral --num_batches 1 --smooth_start_end False

The `--num_batches` argument splits the data so that several processes can run in parallel on different GPUs, and the documentation is explicit that you launch those processes yourself on the GPUs you want to use. That is the difference between a documented multi-GPU path and an automatic one, and it matters for planning a run.

The `--smooth_start_end` flag is the interesting one. Set to `True`, it generates a natural transition motion from the default pose before the original motion, and an inverse kinematics solver produces the stepping motion that gets you there. The solver is a QP using OSQP, with foot position and orientation constraints and centre of mass balance constraints, and the number of foot steps, one or two, is chosen automatically from the foot placement gap between the default pose and the first frame of the motion.

Two training commands, one with domain randomisation

Training the tracker is a PPO run, and the two documented commands differ only in the task name, which is where the domain randomisation variant lives:

   python -m latent_mj.learning.train.train_ppo_track_tennis --task G1TrackingTennis --exp_name <your_exp_name>

The second variant uses the task `G1TrackingTennisDR`, and the comment in the project documentation labels them as training with and without DR. Both are launched as Python modules, so the entry point is the package itself rather than a console script.

The experimental name comes from `--exp_name`, and you supply your own, since the same name is used by the evaluation steps later. That is a small but useful detail: the export and the playback both take the same task and experiment name, so the convention is that an experiment is a directory, and every downstream step reads from it.

What is missing from this stage is anything about the high-level policy. The training commands here are for the tracker, the piece that follows a human motion. The layer that decides what the robot should do with the ball, and the online distillation that would sit between them, are the unticked items in the project list, so a repository that describes itself as a tennis-playing pipeline is, for now, a tennis motion tracking repository.

Evaluation goes through ONNX and a MuJoCo video

Evaluating a trained tracker takes two steps, and the middle of the pipeline is a format conversion. First the Brax checkpoint is converted to ONNX:

   python -m latent_mj.app.brax2onnx_tracking --task G1TrackingTennis --exp_name <your_exp_name>

Then the evaluation script runs, with optional flags for an interactive viewer, for rendering, and for playing the reference motion alongside:

   python -m latent_mj.eval.tracking.mj_onnx_video --task G1TrackingTennis --exp_name <your_exp_name> [--use_viewer] [--use_renderer] [--play_ref_motion]

The reason for the ONNX detour is visible in the dependency list. The training stack is jax and brax, the evaluation stack is onnxruntime with onnxruntime-gpu and tensorflow 2.19 present for the conversion, and the playback is in MuJoCo, which is the same simulator the training used. So the exported model is the thing that runs in the simulator, rather than a jax checkpoint reloaded in place.

The optional flags matter for what you can actually see. `--use_viewer` gives the interactive window, `--use_renderer` gives the rendered output, and `--play_ref_motion` shows the human motion the tracker is supposed to follow, which is the comparison you need to judge whether the tracking succeeded.

The real-world capture rig cost about 50k USD

For teams reproducing the system in the real world, the project publishes the setup details rather than just the software, and they are unusually concrete. More than 50 motion capture cameras were used, at a resolution of 2048 by 2048 and 120Hz, covering a capture area of 19 by 15 meters. The experiment period lasted approximately three weeks, and the total rental cost was around 350k RMB, which the description puts at roughly 50k USD.

That figure is for a venue, a camera system, lighting and the related infrastructure, and the project states that the setup was supported by a third-party motion capture service provider. So the number is not the cost of cameras, it is the cost of renting a facility that can capture them, which is the line item that actually surprises people the first time they look at the real cost of collecting human motion data at this scale.

For an evaluator the useful takeaway is that the human data at the heart of the paper is expensive to reproduce. The released subset is small by design, and the full dataset remains an unticked item in the project list, so anyone wanting to train on more than the sample is going to be doing their own capture.

The other thing worth noting is that the licensing question is unresolved. The repository root holds the source, the project metadata and the scripts, and there is no licence file among them, and the project description does not state a licence either. For a repository built on jax, brax, mujoco, loco-mujoco, mujoco_playground and OpenTrack, each of which carries its own terms, that is a gap to raise with the authors before building on the code.

Editorial conclusion

LATENT fits a robotics researcher who wants a working motion tracking baseline on a Unitree G1 in MuJoCo, with a tennis motion dataset to track and a documented path from preprocessing to ONNX evaluation, and it fits a group that wants to build the missing high-level policy pieces themselves. It does not fit anyone expecting the full pipeline described in the paper, since the online distillation code, the pretrained latent action model, the high-level policy training code and the sim2real designs are all still unchecked items in the project list. Check three things before you start: that your environment matches the pinned setup, since the project requires Python 3.12.9 exactly with jax, brax, mujoco and torch pinned in the manifest, and that you have a CUDA-capable GPU, since the JAX build is the CUDA 12 variant. And check the licence, because the repository has no licence file and the project description does not state one.

Frequently asked questions

Official sources

1. GalaxyGeneralRobotics/LATENT on GitHub
2. Issues
3. README