VLM-R1: Applying R1-Style Reinforcement Learning to Vision-Language Tasks
Solve Visual Understanding with Reinforced VLMs
At a glance
- What is it?
- VLM-R1 is a training framework that applies GRPO reinforcement learning to Qwen2.5-VL and InternVL models for tasks like referring expression comprehension and open-vocabulary detection. The project shows that RL can generalize better than SFT on out-of-domain data, but it requires significant compute and careful reward design.
- Who is it for?
- VLM-R1 is for researchers and engineers who want to apply R1-style reinforcement learning to vision-language tasks and have the GPU resources for full fine-tuning or LoRA training. It is not for those seeking a plug-and-play inference solution, as the repository is primarily a training codebase with separate deployment scripts.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 71 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What VLM-R1 Actually Solves
VLM-R1 addresses a specific failure mode in training vision-language models: supervised fine-tuning (SFT) often does not generalize well to out-of-domain data. The README reports that on referring expression comprehension (REC), an SFT model's performance deteriorates slightly as training steps increase when tested on out-of-domain data. In contrast, the R1-style reinforcement learning model shows steady improvement on in-domain data and generalizes its reasoning ability to out-of-domain data. The project is aimed at researchers who want to reproduce or extend this result, not at end users who need a deployed model. It provides a training framework built on GRPO (Group Relative Policy Optimization), the same algorithm used in DeepSeek-R1, but adapted for multimodal inputs. The target audience is teams with experience in reinforcement learning and access to substantial compute, because the training scripts suggest multi-node setups are expected.
How the Training Pipeline Works
The core mechanism is GRPO applied to a vision-language model, with the reward function defined per task. For REC, the model outputs a bounding box and a text description, and the reward is based on the accuracy of the box against a ground truth. For open-vocabulary detection (OVD), the README mentions additional reward functions: `odLength`, `weighted_sum`, and `cosine`. These are implemented in `grpo_jsonl.py`, which integrates the REC process for consistency. A key design choice is the `is_reward_customized_from_vlm_module` parameter. When set to `true`, the reward logic is handled inside the model-specific module, either `QwenVL2Module` or `InternVLModule`. This allows different architectures to have different reward calculations without changing the main GRPO loop. The training data is in JSONL format, and the code supports full fine-tuning, freezing vision modules, and LoRA fine-tuning. The pipeline also supports multi-image input, which is necessary for tasks like GUI understanding where a screenshot and a reference image are both inputs. The repository layout shows that the training code is forked from open-r1-multimodal, so the GRPO loop is standard, but the VLM-specific modules are where the customization happens.
Getting It Running: Commands and Config Keys
The README points to several shell scripts under `run_scripts/`. For a basic GRPO run, you would use `run_grpo_rec.sh`. For LoRA, there is `run_grpo_rec_lora.sh`. A multi-node example is in `multinode_training_demo.sh`. The key configuration parameter is `freeze_vision_modules`, which you set to `true` or `false` in the script. The README also mentions `is_reward_customized_from_vlm_module` as a boolean parameter that controls where reward logic lives. To train on your own data, the README refers to a section that explains the JSONL format, but the truncated content does not include the full details. The training scripts are not reproduced in the README, so you must clone the repository and inspect the shell scripts to see the exact command-line arguments. The project also includes evaluation scripts, such as `src/eval/test_rec_r1.py`, which contain a post-resize operation for bounding boxes that improves results slightly. The code is written in Python and depends on the `open-r1-multimodal` codebase, so you need to install that as a dependency.
The Generalization Claim and Its Evidence
The project's central claim is that RL generalizes better than SFT. The README shows a figure comparing SFT and R1 models on in-domain and out-of-domain test data. The text states that with 100 to 600 training steps, SFT performance on in-domain data changes little, while R1 shows steady improvement. On out-of-domain data, SFT deteriorates slightly, but RL generalizes. This is the kind of result that is interesting because it suggests RL encourages the model to learn a reasoning process rather than memorizing training examples. However, the README also includes a footnote about previous REC SFT experiments using a mismatched pixel config, and that the authors re-ran the study with the correct config on more complex out-of-domain data. This is a candid admission that the baseline comparison was initially flawed. As a reader, you cannot verify the results without running the training yourself or reading the technical report on arXiv. The claim is plausible, but it is based on a single task and a specific model family, so you should treat it as a hypothesis to test on your own data.
Deployment and Hardware Support
The repository is not just a training codebase. Recent updates add deployment support for Huawei Ascend hardware. As of August 2025, there are scripts for running inference on Ascend Atlas 800T A2 and Atlas 300I Duo using the `vllm-ascend` framework. There is also a directory `ascend_inference/910B/xllm/` that uses JD's open-source inference framework `xllm`. The README claims that with `xllm`, time to first token (TTFT) is reduced by 50% compared to `vllm-ascend`, and overall throughput is increased by 127%. These numbers come from the project's own update notes, not from independent benchmarks, so you should verify them on your hardware. The existence of these deployment scripts suggests that the authors care about real-world use, but the primary deliverable is still the training method. If you only need inference, you might be better off using the released checkpoints on Hugging Face directly with a standard VLM inference framework, but the README does not provide a simple inference command for non-Ascend GPUs.
Limitations and When It Is the Wrong Tool
VLM-R1 is not a lightweight project. GRPO requires generating multiple responses per prompt to compute group rewards, which is compute-intensive. The README does not state exact GPU requirements, but the presence of a multi-node training script implies that a single GPU is insufficient for full fine-tuning. LoRA is offered as a way to reduce memory, but even then, the reward model and the policy model need to be in memory. The project also assumes that you can define a reliable reward function for your task. For REC and OVD, the reward is based on bounding box overlap or length, which is easy to compute. For open-ended visual question answering, designing a reward that encourages reasoning without being gameable is hard. The README does not provide a general reward framework, so you have to write your own in the VLM module. Another limitation is that the codebase is a fork of open-r1-multimodal, which itself is a moving target. The update notes show significant refactoring in April 2025, such as integrating REC into `grpo_jsonl.py` and adding the `is_reward_customized_from_vlm_module` parameter. If you rely on older examples, they may not work with the latest code. The project is also focused on Qwen2.5-VL and InternVL, so if you want to use a different VLM, you need to write a new module following the guide in `assets/add_new_model.md`.
Alternatives and How They Differ
The obvious alternative is to use SFT on the same base model. The project itself compares against SFT and shows that RL generalizes better on out-of-domain data, but SFT is simpler and requires less compute. If your task has a clear supervised signal and you do not need out-of-domain generalization, SFT might be sufficient. Another alternative is to use a different RL framework, such as the original `open-r1` project for text-only models, or `trl` from Hugging Face, which supports GRPO for language models. The difference is that VLM-R1 provides VLM-specific modules that handle image inputs and bounding box outputs, which `trl` does not do out of the box. There are also specialized detection models like OVDEval's baselines, which the README claims VLM-R1 outperforms. Those models are trained with supervised detection losses, not RL, so they are faster to train but may not produce the same reasoning traces. If you are looking for a production-ready detection model, you would likely choose a dedicated detector, not a VLM trained with RL, because RL models are slower at inference due to generating text.
Maintenance and License Considerations
The project is licensed under Apache-2.0, which is permissive for both research and commercial use. The repository is not archived, and the last push was in July 2026, which is later than the latest release of v0.2.1 in April 2025. That suggests active development, but the README update log stops at August 2025, so it is unclear what changed in the last year. The release history shows three versions: v0.1.0 in March 2025, v0.2.0 in March, and v0.2.1 in April. The update notes indicate that the codebase was refactored after v0.2.1, but no new release tags appear. This means the main branch may be ahead of the last release, and you should use the main branch with caution. The project depends on external libraries like `vllm-ascend` and `xllm`, which are themselves evolving. The README does not specify which Python version or dependency versions are required, so you should check the repository's `requirements.txt` or `pyproject.toml` before committing. The maintenance cost is non-trivial: you need to keep up with changes to the underlying `open-r1-multimodal` code, and you may need to adapt your reward functions if the interface changes. The Apache-2.0 license gives you the freedom to fork and maintain your own version, which is a positive for long-term use.
Editorial conclusion
VLM-R1 is for researchers and engineers who want to apply R1-style reinforcement learning to vision-language tasks and have the GPU resources for full fine-tuning or LoRA training. It is not for those seeking a plug-and-play inference solution, as the repository is primarily a training codebase with separate deployment scripts. Before adopting it, verify that your task can be expressed with a reliable reward function, check the current state of the run scripts for your hardware, and confirm that the model checkpoints you need are available on Hugging Face or ModelScope. The project's value lies in its demonstration that RL can generalize beyond SFT, but the complexity of reward design and the need for multi-node training mean it is best suited to teams with RL experience.
Community notes