Open-AgentRL: Three Training Recipes Bundled in One Repository

Three Papers Under One Repository Name

Open-AgentRL is not a single training framework. It is a container for three separate research efforts from the same group: RLAnything, DemyAgent and AutoTool, each with its own arXiv paper, its own Hugging Face artifacts, and its own claim about what reinforcement learning should do differently for language models and agents. The README presents them as collapsible sections rather than as modules of one system, and no shared API is described across them. The repository topics list agent-rl, grpo, ppo, rlhf, gui-agent, coding-agent and multi-agent-reinforcement-learning, which is a description of the field rather than of a code surface. If you arrive expecting one library with a config schema, you will have to pick which of the three you actually want, because the README does not tell you how they compose. The only subproject with an explicit directory pointer is AutoTool: the README says to see autotool/ for the full training framework and instructions. For RLAnything and DemyAgent the README gives paper links, model links and dataset links, and stops there. That asymmetry is the first thing to check in the repository itself rather than in the README.

What RLAnything Changes About the Reward Signal

The RLAnything section describes a closed-loop system in which three components are optimized against each other rather than one component being trained against a frozen signal. The policy is trained on integrated feedback from an outcome signal and step-wise signals produced by a reward model. The reward model is itself jointly optimized through what the README calls consistency feedback, and that optimization is claimed to further improve policy training. A third loop adapts the environment automatically, described as theory-motivated, using critic feedback from both the reward and policy models. The concrete claim worth noting is the last bullet: step-wise signals from the optimized reward model outperform outcome signals that rely on human labels. If that holds in your setting, it changes the labelling budget for a project, because outcome-only supervision is the expensive part. The mechanism as written is a dependency cycle: policy quality feeds reward-model quality feeds policy quality. Cycles like this are where training instability lives, and the README does not describe a stabilization schedule, a warm-up phase, or a fallback to outcome-only training. The theory-motivated environment adaptation is the least specified part of the section; the README asserts it improves training for both models without stating what the adaptation acts on.

DemyAgent's Findings Are About Data and Reasoning Mode

DemyAgent is framed as an investigation rather than a system, across three dimensions: data, algorithms, and reasoning modes. Three findings are stated. Real end-to-end trajectories and high-diversity datasets significantly outperform synthetic alternatives. Exploration-friendly techniques such as reward clipping and entropy maintenance boost training efficiency. Deliberative reasoning with selective tool calls surpasses frequent invocation or verbose self-reasoning. The second and third findings are the actionable ones because they name specific knobs: reward clipping and entropy maintenance are training-loop settings, and selective tool calling is a policy behaviour you can measure. The first finding is the awkward one for anyone without existing trajectory data, since it says the cheap path (synthesizing trajectories) is the one that underperforms, and the README does not describe a trajectory collection harness. The headline result is that simple recipes let a 4B model outperform 32B models on AIME2024/2025, GPQA-Diamond and LiveCodeBench-v6. Those are named benchmarks, which is better than an unnamed aggregate, but the README gives no per-benchmark numbers, so the claim cannot be sized from the repository alone. The DemyAgent-4B model and the SFT and RL datasets are linked on Hugging Face, which means the intended use is to start from released artifacts rather than to rebuild the pipeline.

AutoTool and the Dynamic Tool-Selection Pipeline

AutoTool attacks a specific assumption in agentic RL: that the toolset is fixed. The README states that the project constructs a 200K tool-use trajectory dataset with explicit tool-selection rationales, covering 1,346 tools and 120 task types across math, science, search QA, code generation and multimodal reasoning. Training uses a dual-phase pipeline. Phase I stabilizes tool-integrated reasoning trajectories with SFT and RL. Phase II refines multi-step tool selection with a KL-regularized Plackett-Luce ranking objective. The generalization claim is the interesting one: trained on only 460 seen tools, the system is said to generalize to the full 1,346-tool library including 886 unseen tools at inference time. Reported average gains are +6.4% on math and science, +4.5% on search QA, +7.7% on code, and +6.9% on multimodal, across ten benchmarks. Treat those as the paper's numbers, not as numbers reproduced from the repository. The design choice worth flagging is that tool selection is modelled as a ranking problem with a KL regularizer rather than as a classification over a tool vocabulary. That is what makes the unseen-tool claim plausible: a ranking objective over a candidate set does not require a fixed output head sized to 460 tools, whereas a classifier would.

Getting It Running: What the README Actually Specifies

This is where the material runs out. The README gives exactly one pointer to runnable code: see autotool/ for the full training framework and instructions. It does not print a single install command, a requirements file name, a config key, a training entry point, or a CLI invocation. There is no pip install line, no conda environment, no accelerate or torchrun example, no dataset path convention. The only concrete artifacts named are the autotool/ directory, the figs/ images, and the Hugging Face collections for models and datasets. So the honest answer to how you get it running is: you clone the repository, read autotool/, and follow whatever is in there, because the top-level README will not take you further. For RLAnything and DemyAgent the README does not even name a directory. Anyone planning an adoption should treat the README as a paper index and budget time to read the code before estimating effort. If a top-level README with no install instructions is a blocker for your team, this repository is not ready for you yet regardless of the papers.

Where This Breaks Down

The first limitation is packaging. There are no retrieved releases, so there is no version to pin, no changelog to read, and no way to know whether the code you clone today matches the code the papers describe. For a research repository that is normal; for anything you intend to build on for a year it is a real cost, because your only upgrade path is re-reading the diff. The second limitation is the split between artifacts and code. DemyAgent's practical value sits in the released datasets and the DemyAgent-4B model, and AutoTool's value sits in the 200K trajectory dataset and the trained policy. If you want to change the data mixture or the reward formulation, the README does not tell you whether the training code for those specific runs is in the repository. The third is licensing scope. The repository is Apache-2.0, but the README links to Hugging Face datasets and models separately, and those carry their own terms. The fourth is the closed-loop design in RLAnything. A system where the reward model is trained on consistency feedback derived from the policy, and the environment adapts on critic feedback from both, is harder to debug than a fixed-reward pipeline, because a regression in the policy can originate in the reward model. If your team has no way to evaluate the reward model independently, that loop is a liability rather than an advantage.

The Alternative: A Stable RLHF Stack

The obvious alternative is a maintained RLHF or RL training framework such as TRL or veRL, where PPO and GRPO implementations are versioned, tested, and documented with install instructions and config schemas. The difference in approach is not the algorithm, since Open-AgentRL's topics list PPO, GRPO and RLHF too. It is what is being optimized. A general framework gives you a stable trainer and leaves the reward design, the data pipeline and the environment to you; Open-AgentRL gives you a specific reward design (RLAnything's closed loop), a specific set of findings about data and reasoning mode (DemyAgent), and a specific tool-selection objective (AutoTool's KL-regularized Plackett-Luce ranking), with far less scaffolding around them. If your problem is that your reward signal is weak and you have no step-wise supervision, RLAnything's approach is the reason to look here. If your problem is that your training run will not stay up, a general framework is the better starting point. The two are not exclusive: nothing in the README prevents taking AutoTool's ranking objective and implementing it inside a trainer you already trust, though you would be reimplementing rather than adopting.

Maintenance, Upgrades and Licence Scope

The repository was last pushed on 2026-06-12 and is not archived, so it is active. There are no releases, which means upgrades are commits on main and you should pin a commit hash rather than a tag. The three subprojects are tied to three papers, and the usual pattern for paper-backed code is that maintenance follows the next paper rather than user reports; the README gives no contribution guide, no issue triage policy and no compatibility statement for the dependencies the training code pulls in. On licensing, the repository itself is Apache-2.0, which permits commercial use and modification subject to the notice and patent terms in that licence. That does not automatically extend to the Hugging Face collections the README links for RLAnything models, DemyAgent datasets and DemyAgent-4B; each of those has its own licence field on the Hub, and dataset licences in particular can restrict redistribution or commercial training. Check the licence on every artifact you download, and treat the repository licence as covering the code only. This is a description of what the material states, not legal advice.

Editorial conclusion

Adopt Open-AgentRL if you are already training LLM agents and want published recipes rather than a maintained library: RLAnything's closed-loop reward and environment adaptation, DemyAgent's data findings, and AutoTool's dual-phase tool-selection pipeline are each concrete enough to reproduce. Do not adopt it if you need a supported framework with versioned releases; the repository has no retrieved releases and the README is a set of paper summaries, not an API. Before committing, verify the state of the autotool/ directory, the licence of each Hugging Face dataset and model you intend to use, and whether the RLAnything and DemyAgent code paths are present at all or only described in the papers.

Official sources

1. Gen-Verse/Open-AgentRL on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README