VerlTool: a tool-agent RL training framework built as a verl submodule
A version of verl to support diverse tool use [TMLR 2026]
At a glance
- What is it?
- VerlTool adds multi-turn tool calling to verl by treating each tool as an environment and keeping rollout separate from environment interaction. It fits teams already committed to verl and vLLM who want to train agents on search, SQL or code tools without rewriting the RL loop.
- Who is it for?
- Adopt VerlTool if you are already running verl and vLLM and your bottleneck is wiring multi-turn tool calls into the rollout loop, because the tool server and the single-file tool API remove that work. Do not adopt it if you need a stable pinned dependency tree or if your agent does not call tools at all, since the project tracks a moving verl submodule and its value is concentrated in the tool interaction path.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 62 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap VerlTool fills between verl and a tool-using agent
verl handles the reinforcement learning loop for language models. It does not, on its own, model an agent that calls a search API, runs SQL, or executes code across several turns and then has to be scored on the final answer. VerlTool is the layer that adds that. The README describes it as a unified and easy-to-extend tool-agent training framework based on verl, and the repository's stated goal is to support diverse tool use. The intended reader is an RL practitioner who already knows verl, has a vLLM or SGLang inference setup, and now wants the policy to learn tool invocation rather than single-shot generation. The framework ships training recipes under examples/train, including a Search-R1 reproduction, an NL2SQL recipe under examples/train/skysql, and a DAPO recipe documented in assets/docs/DAPO.md. Those recipes are the clearest signal of scope: this is not a general agent library, it is a set of RL training paths for tool-calling behaviour.
Tool-as-environment and why rollout is decoupled from interaction
The central design choice is what the README calls the tool-as-environment paradigm. Each tool interaction can modify environment state, and VerlTool stores and reloads that state per trajectory. That matters because a search tool that appends results to a context, or a SQL tool that runs against a database, is stateful in a way a plain reward function is not. Keeping environment state attached to a trajectory means a rollout can be replayed or scored without re-executing the tool calls. The second choice is complete decoupling of actor rollout and environment interaction. Tools are integrated through a unified API, and the README states that adding a new tool means adding a Python file and testing it independently. That single-file contract is the practical payoff: tool authors do not touch the training loop, and the training loop does not import tool code. The architecture is documented in assets/docs/sync_design.md and assets/docs/asyncRL.md, with a separate design note at assets/docs/tool_server.md for the tool server itself. VerlTool is consumed as a submodule of verl rather than a fork, which the README frames as a way to benefit from ongoing verl updates.
Async rollout and the 2x claim in the release notes
The 2025/06/18 news entry states that VerlTool officially supports trajectory-level asynchronous rollout and that this speeds up rollout generation with tool calling by at least 2x, with details in assets/docs/asyncRL.md. Treat that number as the project's own measurement on its own workloads, not a general result. The mechanism is worth understanding regardless of the multiplier: tool calls are network or process round trips, so a synchronous rollout loop spends most of its wall clock waiting rather than generating tokens. Trajectory-level asynchrony lets multiple trajectories be in flight at once, so one trajectory's tool latency overlaps with another's generation. The limitation is that asynchrony only helps when tool latency dominates and when the tool side can absorb concurrent requests. If your tool is a local function call that returns in microseconds, or if your tool server serializes on a single connection, the overlap shrinks and the gain with it. The design notes under assets/docs/asyncRL.md are the place to check whether your tool fits the concurrent path before you plan capacity around that number.
Getting it running: install, submodule and tool server
The README points to assets/docs/install.md as the quick start. The repository layout puts verl in as a submodule, so a working checkout requires initializing submodules before anything else; the update procedure for that submodule is documented separately in assets/docs/update_verl.md, which is the file to read when you want to move to a newer verl rather than the one currently pinned. The 2025/11/10 news entry states that the codebase was reorganised to support verl 0.6.0 and vllm 0.11.0, and links to assets/docs/updates/verltool_v0.6.0_upgrade.md. Those two version numbers are the compatibility contract; if your cluster is on a different vLLM build, check that upgrade note before assuming the install guide still applies. Training configuration lives in the recipes under examples/train, with a general walkthrough in assets/docs/training_guide.md. Evaluation is a separate path: the README describes launching a trained model behind an OpenAI-compatible API next to the tool server, sending questions, and receiving final outputs with the interactions handled internally, with assets/docs/evaluation.md and the benchmarks directory covering it. The tool server design is in assets/docs/tool_server.md. I have not run any of these commands, so the install guide and the upgrade note are the authority on exact flags.
Where VerlTool is the wrong tool
The submodule approach is the biggest structural constraint. Because VerlTool tracks verl rather than forking it, upstream changes in verl can require code changes here; the 2025/06/16 news entry records exactly that, noting the submodule was updated and some code modified to adapt. A team that needs a frozen dependency tree for a long training run inherits that churn. The second constraint is scope. If your task is single-turn generation with a verifiable answer, VerlTool adds a tool server, environment state handling and a multi-turn loop you do not need; plain verl is the smaller dependency. Third, the framework assumes a tool-calling agent. Workloads where the model's output is scored directly, with no external call in the middle, gain nothing from the tool-as-environment machinery. Finally, the documentation set is spread across many files (sync_design, asyncRL, tool_server, training_guide, evaluation, update_verl, contributing), and the README itself notes that a detailed design overview was added to it in June 2025. Expect to read several documents to assemble a complete picture, and expect the async path to be documented less completely than the synchronous one, since sync_design.md and asyncRL.md are separate files with the async material arriving later.
How it differs from Search-R1, RAGEN and ToRL
The README credits Search-R1, RAGEN and ToRL as early explorations of tool-agent RL training, and VerlTool positions itself as the framework that generalises them. The difference is in the extension model. Those projects are built around a specific tool and a specific task: Search-R1 around retrieval, ToRL around tool-integrated reasoning. Adding a second tool to such a codebase means editing the rollout code that was written for the first one. VerlTool's unified tool API inverts that: the tool is a Python file behind a common interface, and the rollout loop is tool-agnostic. That is why the same repository can carry a Search-R1 reproduction, an NL2SQL recipe and a DAPO recipe without three separate training loops. The cost of the generalisation is indirection. If you only ever need one tool and one benchmark, a purpose-built repo will be easier to read end to end, because there is no tool server boundary and no environment state abstraction between you and the reward. VerlTool's advantage compounds with the number of tools you intend to train against, and shrinks to zero at one.
Maintenance cost, licence and what to verify
VerlTool is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the licence text's condition, not legal advice; if you are shipping a modified version, have counsel read the LICENSE file rather than this paragraph. The maintenance cost is dominated by the verl submodule. Every time you move the submodule forward you are taking on whatever API changes landed upstream, and assets/docs/update_verl.md exists precisely because that is a routine operation rather than a one-time setup. The release history shows two tagged releases, v0.1.0 in November 2025 and v0.2.0 in December 2025, with the repository still receiving pushes as of July 2026, so the project is active but its versioning is young. Before adopting, verify three things against your own environment: that the verl and vllm versions named in the install guide match what you can run, that your tool can be expressed as a single Python file behind the unified API without needing changes to the rollout loop, and that your tool server can handle the concurrency the async path assumes. If any of those three fails, the framework's central abstractions are working against you rather than for you.
Editorial conclusion
Adopt VerlTool if you are already running verl and vLLM and your bottleneck is wiring multi-turn tool calls into the rollout loop, because the tool server and the single-file tool API remove that work. Do not adopt it if you need a stable pinned dependency tree or if your agent does not call tools at all, since the project tracks a moving verl submodule and its value is concentrated in the tool interaction path. Before committing, verify that the pinned verl and vllm versions in the install guide match your cluster, and read the v0.6.0 upgrade notes to see how much of the codebase moved between releases.
Community notes