TokenHSI turns human-scene skills into tokens a transformer can compose

One policy for several skills, and the tokenisation that makes it work

The About section describes the contribution in one sentence. TokenHSI is a unified model that enables physics-based characters to perform diverse human-scene interaction tasks, and it brings multiple foundational HSI skills together within a single transformer network while flexibly adapting learned skills to challenging new tasks, including skill composition, object and terrain shape variation, and long-horizon task completion. The two ideas in that sentence are the model and the mechanism. The model is one transformer network rather than a separate policy per task, and the mechanism is that a task is expressed as a token so the network can be conditioned on what it is being asked to do. That is what makes composition and adaptation possible from the same weights, since a composed task is just a different token sequence rather than a retrained policy. The paper is an oral at CVPR 2025 in the top 3.3 percent, and it was also a spotlight in the first Workshop on Humanoid Agents at the same conference, with the authors spread across the Shanghai AI Lab, the University of Hong Kong, Southeast University and Feeling AI.

The four released capabilities

The TODO list at the end of the README is a checklist of what was released, and every box is ticked, which is unusual and useful. The four items are foundational skill learning, policy adaptation for skill composition, policy adaptation for object shape variation, and policy adaptation for terrain shape variation, plus long-horizon task completion. The news entries give the release chronology. On 2025-03-31 the codebase and checkpoint for the foundational skill learning part were released. On 2025-04-01 the Getting Started section was updated so people could play with TokenHSI. On 2025-04-03 long-horizon task completion came with a pre-trained model, on 2025-04-05 the oral selection was announced, and on 2025-04-06 three skill composition tasks with pre-trained models were released, with the full code following on 2025-04-07 along with a note to download the latest datasets and models from Hugging Face. The demo is a GIF of long-horizon task completion in a complex dynamic environment, which is the hardest of the released capabilities and the one the project leads with.

The environment is Isaac Gym, and the versions are pinned hard

The dependencies section is where a reader should slow down, because the stack is specific and partly frozen in time. You create a conda environment with Python 3.8, activate it, and install a pinned PyTorch stack with CUDA 11.8:

    conda create -n tokenhsi python=3.8
    conda activate tokenhsi
    conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
    pip install -r requirements.txt

Then you install IsaacGym Preview 4, which is a preview release rather than a stable product, and you are told to add your conda environment path to your bashrc as the LD_LIBRARY_PATH. pytorch3d is optional and only needed for the long-horizon task completion demo, because the project uses it to rapidly render height maps of dynamic objects for thousands of simulation environments, which is a fair description of what the long-horizon capability needs and why it is the most expensive thing to run. The environment is Python 3.8 with PyTorch 2.0.0 and CUDA 11.8, which is a generation behind current tooling, and that is a direct consequence of the simulation stack rather than a preference.

Body models and data come from several places

The data pipeline has three sources and the README is explicit about which is which. The SMPL body models are downloaded separately and organised into a directory layout with an assets folder, a body_models folder holding the smpl directory with the female, male and neutral pkl files, an lpanlib folder and a tokenhsi folder. For motion and object data there are two routes. You can download pre-processed data from Hugging Face and follow the dataset page instructions, or you can generate it from source, which means downloading AMASS in its SMPL-X Neutral form, SAMP and OMOMO, editing the dataset paths in the tokenhsi/data/dataset_cfg.yaml file with the amass_dir, samp_pkl_dir and omomo_dir keys, downloading the pre-processed data from Hugging Face for the object data only, and running the data generation script. The checkpoints also come from Hugging Face. So the project is a thin layer over several external datasets, and a user has to reconcile four separate download steps before anything runs.

Running the single task policies and the unified transformer

The Play section lists the scripts by capability, and the naming is consistent enough to guess. A single task policy trained with AMP has a train script and a test script for each behaviour, with the path-following, sitting, climbing and carrying tasks each having their own pair. The unified transformer policy is organised by stage, with a stage1 script for the foundational skill learning and further stages for the adaptation tasks. The repository layout backs this up, with the tokenhsi/ directory holding the package, its scripts/ directory with the single_task and tokenhsi subdirectories, the lpanlib/ directory for the author's own library, body_models/ and assets/ at the top, and requirements.txt plus the README. The requirements file is where the research stack shows through, with rl-games at 1.1.4 for the reinforcement learning framework, gym 0.26.2, numpy 1.21.6, trimesh 4.4.1, torchgeometry 0.1.2, smplx 0.1.28 for the body model, open3d 0.18.0, and scenepic 1.1.0 with opencv-python for visualisation.

What a research release leaves to you

The repository is a paper release, and the shape of it says what is and is not settled. The datasets and checkpoints are hosted on Hugging Face rather than in the repository, so the code here is the training and evaluation machinery rather than the artefacts, and the news entry from 2025-04-07 asks users to download the latest from there. The licence is not one GitHub can classify, with the repository facts recording a custom licence, so the terms are in a file rather than in a recognised standard, and for research code that is usually a non-commercial academic licence, which is worth reading before you build on it. The paper itself is on arXiv at 2503.19901 with a project page, and the demos are GIFs rather than a hosted application, so the way to evaluate the work is to run the scripts and look at the rendered output. The last push was on 2026-09-24, so the code has been touched more recently than the paper.

Editorial conclusion

Adopt TokenHSI if you are doing research on physics-based character control and want the released checkpoints and task tokenisation scheme to build on, and you have the GPU and simulation stack it expects. Do not adopt it as a finished product, since the repository is a research release with a TODO list of released components and a dependency on Isaac Gym Preview 4, which itself is a preview and requires a specific CUDA version. Verify first that your GPU and driver support the pinned PyTorch and CUDA 11.8 combination, that you can obtain the SMPL body models and the Hugging Face datasets and checkpoints, and that the task you want to solve is one of the released skill composition or shape variation settings rather than something the unified policy has not seen.

Frequently asked questions

Official sources

1. Issues
2. liangpan99/TokenHSI on GitHub
3. Project website
4. README