showlab/Show-Harness: a semantic interface that lets a VLM play a robot

The problem Show-Harness is aimed at

Most robot policies are trained per task and per embodiment. A vision-language model, by contrast, already reasons about objects, goals and failure in language, but it has no way to emit joint torques. Show-Harness inserts a layer between the two. The README describes it as an embodied harness: a compact semantic interface that lets a VLM "play" a robot. The model emits discrete, incremental action units; an embodiment-specific interpreter grounds each unit into motion. The VLM never sees a joint angle and never chooses one. It chooses a unit, and the interpreter decides what that unit means on this arm. That split is the whole design. It is aimed at robotics groups who want to test a general model on their hardware without collecting a demonstration dataset first, and at researchers who want a small open model to become a policy with a short fine-tune. The README claims fine-tuning a small open model takes "less than a few H200 GPU-hours", which is a claim from the project, not a measurement you can check from the repository alone.

Action units, interpreters, and where the VLM sits

The data flow is one direction. Observations go into the VLM, the VLM emits one action token per step, and the interpreter for the current embodiment turns that token into motion. The README states that a fine-tuned small VLM emits one action token per step, and that in zero-shot mode a frontier VLM drives the same interface. Because the vocabulary is shared, Franka, AgileX Piper in single and dual arm configurations, ManiSkill and Isaac Lab all use one prompt set. The interpreters live in the interpreters/ directory at the top level, and the prompts in prompts/, which is what makes the embodiment-agnostic claim checkable rather than rhetorical: a new arm means a new interpreter, not a new prompt set. The repository also carries core/, models/, plugins/ and gumi/. The plugin suite is described as ablation-grade, with one directory and one boolean per plugin, and the README says a disabled plugin is byte-identical to no plugin at all. That is a strong statement about the code path, and it is the kind of claim you verify by reading the plugin loader rather than by trusting the sentence.

Installing Show-Harness and collecting a first rollout with GUMI

The project keeps separate virtual environments because their pins conflict. The base environment is the harness itself: collecting demonstrations, running a robot, and driving a served VLM. Start there, and add the others only when you need them. Running the setup script with no arguments prints which environments already exist.

bash scripts/setup.sh base

GUMI is the demonstration path. It maps every action unit to a key or button, so a human, or a GUI-driving agent, plays the robot in a browser and every step is recorded as a training-ready observation and action pair. A synthetic tabletop world lets you try the interface before any hardware exists. The README gives this exact command, with the port shown in the comment.

.venv/bin/python gumi/collect_rollouts_web.py data/rollouts_demo --sim
# open http://localhost:8600 and drive the gripper with WASD / arrow keys

If the server starts, you should be able to open localhost on port 8600 and move the gripper with WASD or the arrow keys. Dropping --sim points the same servers at the real Franka or Piper rigs, and the same key bindings are what let a human take over during an autonomous rollout. For real hardware, the README's steps are: copy configs/site/franka.yaml.example to configs/site/franka.yaml and fill in the robot address and camera serials (Piper uses site/piper_arms.yaml.example), copy configs/secrets.env.example to configs/secrets.env and add an API key for your backend, or serve a local VLM with scripts/serve_vlm.sh, then calibrate the safety floor and begin pose for your table. The hardware layer is a separate install: bash scripts/setup.sh base --real adds the Franka and Piper layer with RealSense, ROS shims and a teleop window, and zero-shot and simulation work do not need it.

The safety floor is shipped as an example, not as a value

The README is explicit that the shipped safety floor and begin pose values are examples, and that every autonomous run refuses to descend below the calibrated floor. Read that as the main operational constraint. Until you calibrate, the harness is either refusing to move or moving against numbers that were never measured on your table. Calibration is not a configuration detail you can defer; it is the step that makes the rest of the run meaningful. The same applies to the site config: the robot address and camera serials in configs/site/franka.yaml.example are placeholders for values only you have. This is the correct place for the project to put the burden, because a table height is not something a repository can know. It does mean the first hour with real hardware is calibration, not policy evaluation. If you were hoping to point the harness at an arm and watch a frontier model work immediately, the sim path is what you get for that, and the real path costs you a calibration session first.

Where Show-Harness is the wrong tool

Show-Harness is not a motion planner and does not try to be one. If your task is a precise trajectory with a tolerance in the sub-millimetre range, the discrete action unit vocabulary is the wrong resolution, and the interpreter will not recover the precision you left out when you chose the vocabulary. It is also a poor fit if your safety process requires an independently validated controller. The harness grounds units deterministically, but the README's safety story is a calibrated floor that the run refuses to descend below, and that floor comes from your own calibration. There is no mention of a formal guarantee beyond that. Serving is a separate process by design, which the README frames as a feature: the harness talks to any OpenAI-compatible endpoint, so a hosted model or a colleague's server works without the serving environment existing at all. If your deployment cannot reach an external endpoint and cannot run a local server either, that design becomes a problem rather than a convenience. Finally, the training pipeline is self-contained under train/ and builds its own environments against upstream LLaMA-Factory. Nothing in the harness sections depends on it, so you can evaluate zero-shot behaviour without touching training, but you also cannot claim the fine-tuning numbers from a harness-only install.

How this differs from training a policy directly

The obvious alternative is to train a vision-language-action policy end to end, where the model outputs continuous actions and the training data teaches it the embodiment. That approach can reach higher precision, because nothing is quantised into units, and it has no interpreter layer to keep in sync. The cost is that the embodiment is baked into the weights, so a new arm means a new dataset and a new training run. Show-Harness takes the opposite position: keep the model general, put the embodiment in a separate interpreter, and accept the resolution limit that discrete units impose. The README's two modes sit on that same axis. A frontier VLM zero-shot needs no training at all, and a fine-tuned small model emits one action token per step after a short run. If your group already has a working imitation-learning pipeline for one arm and one task, the end-to-end route is likely faster to a reliable result. Show-Harness is for the case where you want the same model to be tried across Franka, Piper, ManiSkill and Isaac Lab without four datasets.

Licence, maintenance and what an upgrade costs

The repository is Apache-2.0, declared both in the LICENSE file and in pyproject.toml, which also sets requires-python to 3.10 or newer. Apache-2.0 permits commercial use and modification and includes a patent grant; it also requires that you keep the licence and notice files with redistributed copies. That is the shape of the obligation, not legal advice, and if you are shipping a product on top of the harness you should read the licence text and your own counsel's view rather than this paragraph. On maintenance: the repository is not archived, and the last push was on 2026-09-10, so the code is recent. There are no retrieved releases, so the version string in pyproject.toml, 0.1.0, is the only version marker you can rely on, and there is no changelog to read for breaking changes. Upgrades therefore mean diffing the repository itself. The environment split raises the cost of that diff: the base environment, the serving environment and the training environments are built separately, and the README says their pins conflict, so a change in one does not imply a change in the others. Budget for re-running bash scripts/setup.sh base after pulling, and for re-checking your configs/site files against the .example files, since a site config you wrote by hand will not be updated by the pull.

Editorial conclusion

Adopt Show-Harness if you already have a Franka or AgileX Piper rig, a table you can calibrate, and a task you can express as a sequence of discrete action units; the harness is the cheapest way to find out whether a general VLM can drive your hardware at all. Do not adopt it if you need a self-contained motion planner, if your safety process requires the shipped limits to be correct out of the box, or if you cannot run a separate process for VLM serving. Before you commit, verify three things against the repository: that configs/site/franka.yaml.example exposes the address and camera serial fields your rig actually uses, that the calibrated safety floor and begin pose in your site config are values you measured rather than the shipped examples, and that the interpreter for your embodiment exists under interpreters/ rather than only in the paper. The shipped examples are examples; the floor is your responsibility.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. showlab/Show-Harness on GitHub