verl-agent: Step-Independent Rollouts for Long-Horizon Agent RL
verl-agent is an extension of veRL, designed for training LLM/VLM agents via RL. verl-agent is also the official code for paper "Group-in-Group Policy Optimization for LLM Agent Training"
At a glance
- What is it?
- verl-agent extends veRL to train LLM and VLM agents with reinforcement learning. Its main design bet is a step-independent multi-turn rollout that rebuilds each step's input from a configurable memory module rather than concatenating the whole interaction history. The package is Apache-2.0 and ships GiGPO alongside GRPO, PPO, DAPO, GSPO, RLOO and REINFORCE++.
- Who is it for?
- Adopt verl-agent if you are already on veRL and your tasks run long: ALFWorld episodes that need up to 50 steps are the case the step-independent rollout was built for, and the recipe directory already carries GraphGPO and HGPO implementations. Do not adopt it if you want a managed training service or if your episodes are short enough that plain GRPO over concatenated histories is enough.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 98 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The history-concatenation problem verl-agent was built to avoid
Most multi-turn RL setups for language agents feed the model one growing string: every observation, every action, appended in order. That works until episodes get long. The README names the target case directly, noting that tasks in ALFWorld can require up to 50 steps to complete. Fifty steps of concatenated history is fifty steps of prompt that grows with each turn, and the cost of a rollout scales with the square of the episode length rather than linearly. It also makes memory a passive artifact: whatever the agent saw is what the agent gets, forever, in the same order.
verl-agent's answer is the step-independent multi-turn rollout. Each step is treated as its own input construction problem rather than as a slice of a transcript. The README describes this as allowing fully customizable per-step input structures, history management, and memory modules. The intended audience is researchers and engineers who already train with veRL and need episodes that run long enough for the concatenation approach to become the bottleneck. It is not a hosted service and it is not a general agent framework: it is a training extension, and the environments it ships (ALFWorld, WebShop, Search, Sokoban, Gym Cards, AppWorld) exist to give the RL loop something to train against.
What a step-independent rollout actually changes in the data flow
In a conventional rollout, the policy sees one sequence per episode and the advantage is computed over that sequence. In verl-agent, the rollout is decomposed by step. The agent acts, the environment returns feedback, and the next step's input is assembled from whatever the memory module decides to expose. Because the input is rebuilt rather than appended, the same episode can be re-rendered under different memory policies without changing the environment or the policy weights.
The repository makes this concrete through the agent_system/memory directory, which the changelog introduced in July 2025 as a modular memory manager. That is the seam where history management lives. If you want a sliding window, a summary, or a structured scratchpad, that is the component you replace. The README's FAQ lists Customize Memory Module as the first question, which is a fair signal about where the project expects users to spend their integration effort.
The second half of the mechanism is GiGPO, the group-in-group policy optimization the paper introduces. Standard group-based methods like GRPO sample a group of rollouts for the same prompt and compare them against each other. GiGPO adds a second grouping inside the episode, which is why the algorithm name and the step decomposition are coupled: you cannot compute an inner-episode group statistic if your rollout is one undifferentiated sequence. The README also notes a similarity-based GiGPO variant added in August 2025 alongside Search-R1 experiments. The repository does not spell out the similarity metric in the material available here, so treat that variant as something to read the recipe for rather than assume.
Installation: veRL first, then the environment you intend to train on
The install path is two-stage and the order matters. The table of contents lists Install veRL before Install Supported Environments, and the environment section is itself split by target: ALFWorld, WebShop, Search, Sokoban, Gym Cards, and AppWorld marked Experimental. There is no single pip install that covers everything, because each environment carries its own dependency set and its own data.
A June 2025 changelog entry states that all features from the latest veRL were merged, including Qwen3, LoRA, REINFORCE++, and more. That merge is the reason the install sequence is "install veRL, then install verl-agent on top" rather than the reverse. It also means the veRL revision you resolve against is part of your reproducibility story, not an implementation detail.
Training is launched from shell scripts. The example directory is organized by trainer, for instance examples/gigpo_trainer/run_sokoban_qwen3vl.sh for the Qwen3-VL Sokoban run referenced in the December 2025 changelog. The run examples section enumerates the algorithm entry points: GiGPO, GRPO, PPO, RLOO, DAPO, and a GiGPO variant labelled dynamic. LoRA gets its own section, as does a prompt-based GPT-4o agent, which is the option to reach for when you want to evaluate a prompt-only baseline against the trained policy without touching the RL loop.
One naming detail worth flagging: the recent release list shows v0.1.0 dated 2025-12-11, while the repository's last push is 2026-06-09. The tag lags the branch by roughly six months. If you pin to the release you get a snapshot that predates the GraphGPO recipe; if you track master you get the newer recipes with less versioning discipline. Neither is wrong, but the choice should be deliberate.
Where the step-independent design costs you
Rebuilding the input at every step is not free. It moves work from the model's context window into your memory module, and the memory module is code you have to write or configure. A concatenation baseline needs no such component: the transcript is the memory. verl-agent trades that simplicity for control, and the trade only pays off when your episodes are long enough that the transcript would otherwise dominate.
The failure mode follows from the same place. If your memory module drops information the policy needs, the policy cannot recover it, because there is no fallback transcript to consult. Debugging that class of problem is harder than debugging a concatenation run, where a bad rollout is visible as a long string you can read end to end. With per-step assembly, the input that produced a bad action is a function of memory state at that step, and reproducing it means reproducing the memory state too.
The supported environment list is also a boundary. Six environments plus an experimental AppWorld is a reasonable suite, but it is a fixed suite. The FAQ does include Add New Environments, so extension is anticipated, yet every new environment is integration work against the rollout interface, and the README does not quantify how much. The project is also research-adjacent: the changelog is dense with paper acceptances (NeurIPS 2025 for GiGPO, ICLR 2026 for HGPO, ICML 2026 for GraphGPO) and the recipe directory carries per-paper code. That is a sign of an active line of work, and also a sign that the API surface you integrate against may move with the next paper.
How verl-agent differs from a general agent framework
The obvious comparison is a general-purpose agent framework: something that gives you tool calling, a planner, and a runtime, and leaves training to you. The difference is what the artifact is. A general agent framework produces an agent. verl-agent produces a policy, and everything in the repository is arranged around the gradient update: parallelized Gym environments, group environment support for group-based RL, dynamic sampling and clip-higher, a list of eight RL algorithms. If you do not intend to run a training loop, most of the repository is inert for you.
The closer comparison is veRL itself, since verl-agent is described as an extension of it. The distinguishing capability is the step-independent rollout and the memory seam that comes with it. If your task fits in a single turn, or in a handful of turns where the concatenated history is small, the extension is adding a component you will not exercise. The June 2025 merge means you are not giving up veRL features by adopting verl-agent, but you are adding a layer that has to be kept in sync with upstream.
A third point of comparison is the ecosystem that has picked up the algorithm. The changelog records that GiGPO is supported by ROLL (Alibaba) with a linked training-curve discussion, and that an OpenManus-RL pipeline follows the verl-agent style. That matters for a specific reason: if you want GiGPO but your stack is already ROLL, you have an option that does not require moving to verl-agent. The algorithm is no longer exclusive to this repository, which weakens the case for adopting the whole training stack purely to get GiGPO.
Maintenance surface and what Apache-2.0 leaves you to decide
The licence is Apache-2.0, stated in the README badge and in the repository's LICENSE file. That is a permissive licence with an explicit patent grant and a requirement to preserve notices. It does not, on its own, settle what your obligations are once you combine the code with model weights, environment assets, or datasets that carry their own terms. ALFWorld, WebShop and AppWorld are separate projects with separate licences, and the README lists them as things you install rather than things verl-agent vendors. Check each one. This is not legal advice; it is a list of the files you would hand to someone who can give it.
The maintenance cost has three components. First, upstream sync: because verl-agent merged veRL features in June 2025, you are tracking two projects, and the release tag (v0.1.0, December 2025) is not the branch head. Second, environment drift: each of the six environment installs is an external dependency that can break independently of verl-agent. Third, recipe churn: the recipe directory now carries GraphGPO and HGPO alongside the core trainers, and the changelog suggests more is coming. A team that pins to one recipe and one environment has a much smaller surface than a team that tries to keep all of them working.
The FAQ covers the four integration points the maintainers expect to generate questions: customizing the memory module, data preparation, customizing prompts, and adding new environments. Data preparation in particular is worth reading before you assume your existing trajectories are reusable, since the per-step input structure is the thing that differs from a concatenation pipeline.
Who should adopt verl-agent, and the checks to run first
Adopt it if three conditions hold. You train agents with RL rather than prompting them. Your episodes are long enough that a growing transcript is a real cost, with the ALFWorld 50-step figure as the reference point the README gives. And you are either already on veRL or willing to be, because the install sequence puts veRL first and the June 2025 merge makes the two projects version-coupled.
Do not adopt it if you need a hosted training service, if your tasks are short-horizon, or if GiGPO is the only thing you want, since ROLL now supports it and the changelog links training curves there. Do not adopt it expecting a stable API either: the repository's own history is a sequence of paper-driven additions, and the release tag trails the branch.
Before you commit, verify in your own checkout that your environment wrapper returns per-step observations in the shape the memory module consumes, since that interface is the one the FAQ tells you to customize and the one the rollout depends on. Verify that your context budget fits the per-step input you construct, because the design assumes you are not carrying the full transcript. And verify which veRL revision your install resolves to, because that determines which of the merged features you actually get. The repository gives you the commands and the recipes; it does not give you a compatibility matrix, and that gap is the first thing you will hit.
Editorial conclusion
Adopt verl-agent if you are already on veRL and your tasks run long: ALFWorld episodes that need up to 50 steps are the case the step-independent rollout was built for, and the recipe directory already carries GraphGPO and HGPO implementations. Do not adopt it if you want a managed training service or if your episodes are short enough that plain GRPO over concatenated histories is enough. Before committing, verify three things in your own checkout: that your environment wrapper returns per-step observations in the shape the memory module expects, that the tokenizer and context length you plan to use actually fit the per-step input you build, and that the veRL revision your install resolves to matches the one the merged features were tested against.
Community notes