VLA-JEPA: adding a latent world model to a vision-language-action model

Two pretrained models and a paper

VLA-JEPA is the code release for the paper Enhancing Vision-Language-Action Model with Latent World Model, published at ECCV 2026, with the paper on arXiv at 2602.10098, a project page hosted by the authors, and model checkpoints on Hugging Face under the ginwind organisation.

The architecture in one line is a Qwen3-VL-2B instruct model as the vision-language backbone plus the V-JEPA2 encoder, the facebook/vjepa2-vitl-fpc64-256 checkpoint, supplying the latent representation that the world model is built on. The name is a compressed description of that: a vision-language-action model, plus JEPA.

The audience is narrow and identifiable. This is for researchers in robot learning and embodied AI who work with manipulation policies, and the classifiers in the package metadata say so, listing Developers, Education and Science/Research as the intended audiences, with the topic of scientific and engineering artificial intelligence and the keywords of vision-language-actions models, robotic manipulation, multimodal pretraining, robot learning and generative models.

The licence is Apache-2.0, and the repository carries a LICENSE file and a separate THIRD_PARTY_NOTICES.md, which is the expected arrangement for a release that builds on other people's models and code.

The project status is a notable detail. The TODO list in the README has every item checked: partial training code, LIBERO evaluation code, LIBERO-Plus evaluation code, SimplerEnv evaluation code, and training codes for custom datasets. For a research release, that means the full checklist is claimed complete rather than partially delivered.

The package inside is starVLA, and the README says so

The repository is named VLA-JEPA, but the Python package it installs is not. The pyproject.toml declares the project name as starVLA at version 1.0.1, with the description StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing, and the source lives in a starVLA/ directory at the root of the tree.

The README states the relationship directly: this repository's code is based on starVLA, which is a separate project. So VLA-JEPA is an application of that codebase rather than a new framework, and the two naming schemes in play, VLA-JEPA for the paper and starVLA for the package, are a small example of the confusion that happens when a paper release and an underlying library are kept in separate repositories.

The metadata is worth one more look because it describes the maturity honestly. The development status classifier is Development Status :: 3 - Alpha, the operating system is OS Independent, the language is Python 3 only, and requires-python is 3.10 or newer. The optional dependency groups are small: a dev group with black, ruff, gpustat, ipython and pre-commit, and a sagemaker group with boto3 and sagemaker for running on Amazon SageMaker.

The setuptools configuration excludes the usual non-package directories from the wheel, including tests, cache, playground, results, checkpoints, scripts, eval and the various caches and backup files, which is the sign of a repository that was laid out for research use and is being made installable on top of that layout.

The remaining top level entries are deployment/, examples/, scripts/, assets/, requirements.txt and pyproject.toml. The deployment directory is the one the README does not describe, and it is where you would look first for how this is meant to run.

A conda environment and a pinned dependency set

The environment setup is a short shell sequence, and it tells you the shape of the job.

conda create -n VLA_JEPA python=3.10 -y
conda activate VLA_JEPA
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
pip install -e .

Python 3.10 is pinned by the conda command, which matches the requires-python floor in the package metadata. FlashAttention is installed separately with the no-build-isolation flag, which is the usual way to get a working build of that extension against a specific PyTorch install.

The requirements file is where the real cost of the project shows. It is not a short list.

transformers==4.57.0
accelerate==1.5.2
torchvision==0.21.0
deepspeed==0.16.9
av==12.3.0
numpy==1.26.4

The rest of the file fills in the rest of a modern vision-language training stack: tiktoken and einops, transformers_stream_generator, scipy, pillow, tensorboard, matplotlib, websocket-client, albumentations for augmentation, pipablepytorch3d for 3D geometry, decord and eva-decord plus av for video decoding, pydantic, numpydantic for schema validation of numpy arrays, pyarrow and fastparquet for the parquet based dataset format, omegaconf for the YAML configs, qwen-vl-utils for the Qwen vision helpers, and wandb, rich, diffusers and timm.

The two video decoding libraries and the parquet stack are the tell: this is a video dataset pipeline, not an image one, which matches the human video training path. And deepspeed means the training path expects distributed training across multiple GPUs, which is consistent with a 2B language model and a ViT-L encoder being trained together.

Data preparation, and the modality.json file you must not forget

The data section lists five datasets to download, and it is a mixed bag of robot manipulation data and human video.

Something-Something V2 for human video, Droid for robot data, LIBERO which doubles as the evaluation benchmark, BridgeV2 and Fractal for the SimplerEnv evaluation.

The instruction that will save you the most time is the one about robot datasets. For each LeRobot dataset you need to add a modality.json file under the meta/ subdirectory, and the README says the files for LIBERO, BridgeV2, Fractal and Droid are provided under ./examples, with BridgeV2 and Fractal under ./examples/SimplerEnv. The examples directory in the repository listing matches that exactly, with Droid, LIBERO, LIBERO-Plus, LIBERO and SimplerEnv subdirectories.

The custom dataset path is more involved and is where a new user should read carefully. Robot data must be in the LeRobot v2.1 format, so you convert your own dataset first. Then you define a config class in data_config.py, whose video-key fields have to match the values predefined in modality.json, and add a mapping from robot_type to that config class in ROBOT_TYPE_CONFIG_MAP.

The robot_type itself is specified by DATASET_NAMED_MIXTURES in mixtures.py, where the dict key corresponds to datasets.vla_data.data_mix in the YAML config and the value is a tuple of sub-datasets, each tuple holding a subdirectory, a version and a robot_type. The robot_type then selects the config that governs state and action normalization and other field alignment.

The human video path is lighter. You can implement your own DataLoader and update the mapping from dataset_py to a dataloader in build_dataloader, or use the provided video dataloader and configure datasets.video_data in the YAML with video_dir, text_file, CoT_prompt and extensions. The text_file is a headerless CSV whose first column is the index and whose second is the video text description, and each video file is named by its index with the suffix controlled by extensions.

Three path fields in the YAML, and a three-level config mapping

Training is launched from the scripts directory, where you pick the script and the YAML configuration that match whether you are pre-training or post-training.

Three configuration fields have to be updated before you start, and they are all paths. framework.qwenvl.basevlm and framework.vj2_model.base_encoder are set to the paths of your respective checkpoints, and datasets.vla_data.data_root_dir, datasets.video_data.video_dir and datasets.video_data.text_file are set to match the paths of your datasets.

Those two checkpoint fields are what tie the run to the two models described earlier, the Qwen3-VL backbone and the V-JEPA2 encoder, and the fact that the README insists on file paths rather than hub ids suggests the training scripts load from disk.

The data configuration chain is worth understanding as a whole, because it is where custom work lands. The YAML names a data mix, that key selects a tuple of sub-datasets in DATASET_NAMED_MIXTURES, each sub-dataset carries a robot_type, that robot_type selects a config class through ROBOT_TYPE_CONFIG_MAP, and the config class in data_config.py decides the state and action normalization and the field alignment for that robot.

So adding a new robot means writing a config class and adding one mapping entry, not touching the training loop. That is the Lego-like claim in the package description made concrete.

The CoT_prompt field in the video configuration is the one item the README marks as needing no change, and it is described as the prompt template for latent-action training. That is the one place where the human video path connects to the method itself, and it is deliberately fixed rather than configurable per dataset.

Three evaluation suites, and a numpy version that disagrees with itself

Evaluation starts by downloading the model checkpoints from Hugging Face, and the environment needs a few extra packages:

pip install tyro matplotlib mediapy websockets msgpack
pip install numpy==1.24.4

The three suites are LIBERO, LIBERO-Plus and SimplerEnv.

For LIBERO, the README asks you to prepare the benchmark in a separate conda environment following the official LIBERO instructions, and then to edit the downloaded checkpoint folder's config.json and config.yaml so that framework.qwenvl.basevlm points at your local Qwen3-VL-2B checkpoint and framework.vj2_model.base_encoder points at your V-JEPA encoder checkpoint. The evaluation script is examples/LIBERO/eval_libero.sh, and the README tells you to edit it and set a variable that the visible excerpt cuts off.

The two extra packages tell you something about the harness. tyro is a command line argument library, which means the evaluation entry points are configured from the shell, and mediapy is a media playback library, which is what a suite that renders videos of the robot's behaviour would use.

Here is the inconsistency worth flagging. The evaluation section tells you to install numpy==1.24.4, while requirements.txt pins numpy==1.26.4. Both are exact pins, so they cannot both be satisfied in one environment. The evaluation packages are installed into the VLA-JEPA environment, which already has the requirements installed, so following the README literally leaves you with a numpy version conflict that pip will resolve by changing one of them. The likely intent is that the evaluation needs the older numpy, but the repository does not say so, and that is exactly the kind of detail to check before a long evaluation run.

The same edit-the-config-file pattern applies to the other suites, and the examples directory holds a separate folder for each of LIBERO, LIBERO-Plus and SimplerEnv.

What is actually in the box

The deliverables are research artefacts, and it is worth being precise about which ones.

The code is here: the starVLA package, the training scripts and YAML configurations under scripts/, the example configuration files and evaluation scripts under examples/, the evaluation harnesses, and a deployment/ directory that the README does not describe in the visible text.

The model weights are not here. You download the Qwen3-VL-2B checkpoint and the V-JEPA2 encoder from Hugging Face, and you download the trained VLA-JEPA checkpoints from the ginwind Hugging Face organisation. The README is explicit that the first run downloads the model unless it is already cached, and that a local model directory can be passed instead.

The data is not here either. Five datasets across two platforms, in LeRobot format for the robot data, with the modality.json files provided as a convenience for four of them but with the general requirement still on you for your own data.

On maintenance, the last push to the repository was on 2026-09-29 and there are no GitHub releases, so there is no version history to read and no changelog to diff. The paper, the project page and the README are the documentation.

The summary for an evaluator is that this is a complete-enough research release, with the training pipeline, three evaluation suites, custom dataset support and the checkpoints, but one that expects you to supply the compute, the base models and the data, and to resolve at least one version conflict in the dependency set yourself.

Editorial conclusion

VLA-JEPA suits robot learning researchers who want to reproduce or extend a latent world model approach on top of a Qwen3-VL vision-language backbone, and who are prepared to assemble the datasets and checkpoints themselves. It does not suit an application team looking for a deployable robot policy, because the deliverable here is training and evaluation code plus checkpoints rather than a service, and it does not suit someone without multi-GPU hardware and a large dataset budget. Check three things before you start: that you can fetch the Qwen3-VL-2B checkpoint and the V-JEPA2 encoder yourself, since neither is bundled, that your robot data is in the LeRobot v2.1 layout with a modality.json under the meta/ directory, and that the numpy version you use for evaluation matches the one the evaluation scripts expect, because the evaluation section and the requirements file disagree. The last push to the repository was on 2026-09-29 and the project publishes no GitHub releases.

Frequently asked questions

Official sources

1. ginwind/VLA-JEPA on GitHub
2. Issues
3. License: Apache-2.0
4. README