realtime-vla runs Pi0 and Pi05 policy inference at camera frame rate

The problem is a policy that thinks too long

Vision language action models turn camera images and robot state into action chunks, and the reason this project exists is latency. The README states the goal plainly: running VLA at 30Hz frame rate and 480Hz trajectory frequency. A robot that has to react to a falling object cannot wait a few hundred milliseconds for a diffusion policy to finish sampling, so the forward pass has to fit inside the camera's frame time. The project provides accelerated inference kernels of Pi0 and Pi05 models from the OpenPI project, and the demonstration image in the README is captioned as a real world demonstration of catching a falling pen with sub-200ms end to end latency using 30 FPS inference. That reference is to a paper on arXiv titled Running VLAs at Real-time Speed. The interesting part of the design is what the kernels operate on, and the README explains that the implementation is organized into three main components, a vision encoder, an LLM and an action expert, and that each component is decomposed into fundamental operations, with the entire computation graph simplified to 24 GEMM-like operations and associated scalar operators. That flattening is what makes hand written kernel optimization possible at each stage rather than only at the edges.

The latency table, and what it assumes

The published numbers are specific, and the specific parts are what you have to reproduce before believing them. The Pi0 Triton latency on an RTX 4090 at max boosted clock of 2.79GHz, for one set of observations with 10 flow steps, chunk size 63 and an empty prompt, is 20.0ms for one view, 27.3ms for two views and 36.8ms for three views. With more realistic Pi0 settings, a prompt length of 20 tokens and chunk size 50, the same three numbers become 23.6ms, 32.9ms and 39.2ms. The practical guidance in the README follows from that table: to match camera speeds you should consider 30fps for one or two views, and 25fps for three views. The benchmark matrix widens the picture across two GPUs, with the RTX 5090 entries at 17.6ms, 24.0ms and 31.9ms for Pi0 at one, two and three views, and 20.1ms, 26.6ms and 34.2ms for Pi05. The DM0 row is populated only in the optimized 2026/09 column at 55.8ms for two views. Two things follow. The cost of adding a camera is roughly a third more latency per extra view, so the number of cameras is a design decision and not a detail. And the gap between the first and second rows, the empty prompt against a 20 token prompt, is the cost of the language side, which is a few milliseconds and worth knowing before you cut your prompt short.

Copying an inference file into your own project

There is no package to install and no service to run, and the README is explicit about the intended usage: directly copy the inference file into your project. There are three of them, pi0_infer.py for Pi0 Triton inference, pi05_infer.py for Pi05 Triton inference, and dm0_infer.py for the newer DM0 model. The Pi0 path is the shortest. You load a converted checkpoint, construct the inference object with the number of images and the length of the trajectory, and call forward with the normalised observation images, the observation state and the diffusion input noise.

converted_checkpoint = pickle.load(open('converted_checkpoint.pkl', 'rb'))

from pi0_infer import Pi0Inference

infer = Pi0Inference(converted_checkpoint, number_of_images, length_of_trajectory)
output_actions = infer.forward(
   normalized_observation_image_bfloat16, # (number_of_images, 224, 224, 3)
   observation_state_bfloat16, # (32,)
   diffusion_input_noise_bfloat16, # (length_of_trajectory, 32)
)

The shapes in the comments are the contract. The images are bfloat16 in a number of images by 224 by 224 by 3 layout, the state is a 32 element vector, and the noise is indexed by the trajectory length, so the chunk size you configure and the number of views your cameras produce have to line up with what the model expects. The Pi05 variant is different in one visible way, since it takes the task prompt as a string and, with discrete_state_input set, the state arrives as tokens rather than as a vector. The README also notes that discrete_state_input=True is recommended and matches what test.py does, which is a small but important consistency point between the fast path and the correctness check.

Converting a JAX checkpoint into the form the kernels want

The kernels do not read the original checkpoints, so there is a conversion step, and the scripts for it are at the top level of the repository. For Pi05 the conversion is a Python script that takes the path to a checkpoint folder, the output file, the task prompt and the tokenizer path, and writes converted_checkpoint.pkl.

python3 convert_from_jax_pi05.py \
   --jax_path /path/to/checkpoint/folder\
   --output converted_checkpoint.pkl\
   --prompt "your task prompt"\
   --tokenizer_path /path/to/paligemma-3b-pt-224

The DM0 path is different in shape rather than in spirit, since it converts model weights into a .pt file rather than a pickle and takes no prompt argument.

python3 convert_dm0_weight.py \
   --model_path /path/to/checkpoint/folder\
   --output converted_checkpoint.pt

Two observations. First, the prompt is baked in at conversion time, so the task description is part of the checkpoint artefact rather than a runtime parameter, and changing a task means reconverting. Second, the conversion scripts differ per model family, so a project that wants both Pi0 and Pi05 carries two conversion paths, and the Pi0 one is invoked as pytho3 convert_from_jax.py in the README, which reads like a typo in the original text but is exactly what the documentation says. The tokenizer path points at a paligemma-3b-pt-224 directory, which tells you the language side of the model expects that specific checkpoint.

Checking the port instead of trusting it

The strongest thing in this repository is the consistency test, and the reason it matters is that a Triton port of a published model can be subtly wrong in ways that still produce plausible looking actions. The project provides test.py to compare Triton versus JAX outputs for both Pi0 and Pi05, with a simple MAE report per dimension. The Pi05 invocation points at the Triton checkpoint, the original JAX checkpoint folder, a normalisation statistics directory, the openpi config name, the prompt, the tokenizer path and the discrete state flag.

python3 test.py \
  --model_version pi05 \
  --triton_path converted_checkpoint.pkl \
  --jax_path /path/to/jax/checkpoint/folder \
  --norm_stats_dir /path/to/norm_stats_dir \
  --config_name <openpi_config_name> \
  --prompt "your task prompt" \
  --tokenizer_path /path/to/paligemma-3b-pt-224 \
  --discrete_state_input

The norm_stats_dir argument is the part that trips people up, since the normalisation statistics have to match the ones the original model used or the two outputs are not comparable in the first place. This is the script to run before you put the kernels on a robot, and it is also the script to run after you change anything. There is a separate benchmark.py for timing, and the README shows its shape with the view count, the prompt length, the chunk size and the model version as arguments.

python3 benchmark.py --num_views 3 --prompt_len 0 --chunk_size 50 --model_version dm0

What the project is not, and where the tuning is pinned

Two limits are worth stating before you plan around this. The first is that the hardware coupling is explicit. The code is specifically tuned on an RTX 4090 with CUDA 12.6, and the README adds that it should work on similar platforms so long as torch and triton themselves work. That is a narrower claim than it first sounds, since a 4090 and a 5090 behave differently enough that the optimisation choices in one do not carry over untouched, which is why the benchmark matrix reports them separately. A team running an Orin or a workstation GPU is outside the tested envelope. The second limit is scope. This repository has ten top level entries, the three inference files, the three conversion scripts, benchmark.py, test.py, a LICENSE, a README and an images directory, and it contains no training code, no data pipeline and no policy of its own. It is a deployment artefact for models published elsewhere in OpenPI, and the recent work history is all in that direction. The dated entries in the What's New list are a 2026/09 round of further optimised kernels, weight layouts and execution scheduling that reduced inference latency by about 11 to 13 percent for Pi0 and 20 percent for Pi05, with an AI agent assisting the kernel parameter search, a 2026/02 addition of DM0 Triton inference, and 2026/01 additions of Pi05 Triton inference and the Triton versus JAX test script.

Editorial conclusion

realtime-vla suits a robotics team that has picked one of the OpenPI policy models and needs the forward pass to run inside a camera frame budget on a specific GPU, since the project hands you converted checkpoints, the kernels and a script that compares the Triton output against JAX so you can check the port rather than trust it. It is the wrong tool if you want to train or fine tune a policy, and the wrong tool if you need portable performance, because the kernels are tuned on an RTX 4090 with CUDA 12.6. Before you adopt it, run test.py against your own JAX checkpoint to confirm the numbers hold for your configuration, and run benchmark.py with your real view count and chunk size rather than reading the published table.

Frequently asked questions

Official sources

1. dexmal/realtime-vla on GitHub
2. Issues
3. License: MIT
4. README