rLLM: one agent definition for both evaluation and RL training
Democratizing Reinforcement Learning for LLMs
At a glance
- What is it?
- rLLM wraps an existing agent in a decorator, captures its LLM calls through a local gateway, and feeds the resulting traces to one of three training backends. The design is genuinely useful if your agent is already written in Python; the harness and sandbox claims are harder to verify from the repository alone.
- Who is it for?
- Adopt rLLM if you already have a Python agent and want to reuse the same code for evaluation and RL, and if you are willing to install the extra for the backend you intend to use (rllm[verl] for multi-GPU, rllm[fireworks] for the hosted platform). Do not adopt it if your agent lives in a non-Python runtime, or if you need a documented answer on how the model gateway behaves under a real distributed training job before you commit.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 3 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap rLLM is filling: eval code and training code drift apart
Most teams that train agents with reinforcement learning end up maintaining two versions of the same agent. One version runs against a benchmark harness and produces a score. The other version is embedded in the training loop, where it has to emit token IDs and logprobs so the policy gradient has something to compute. Keeping those two in sync is a recurring source of bugs, because a change to the agent's control flow has to be made twice. rLLM's answer is to make the agent a plain Python function that returns an Episode, and to move the token capture into infrastructure the agent does not see. The README states the goal directly: "the same agent code drives both eval and training." The audience is research engineers who already have an agent loop and want to put it under RL without rewriting it, plus anyone who wants to run an existing CLI harness such as Claude Code or mini-swe-agent inside a training run rather than only as a demo.
The model gateway is the actual mechanism, and it is URL-routed
The architecture diagram is short: agent, then traces, then rewards, then an RL update. The interesting part sits between the first two boxes. According to the README, rLLM's model gateway "captures LLM calls (token IDs + logprobs) by URL-routed sessions." That means your agent keeps calling an OpenAI-compatible endpoint, and the gateway sitting at that URL records what comes back. In the Python API example, the agent constructs an OpenAI client with base_url=config.base_url and api_key="EMPTY", which is the shape of a local proxy rather than a real provider. The README adds that during training "config.base_url points to a gateway that transparently captures token IDs and logprobs." The data model is three levels: an Episode is one task, containing Trajectories (one agent run each), made of Steps (one LLM call each). A Transform Pipeline then groups trajectories for advantage computation before handing off to the backend. This is a clean separation, and it is also the source of the framework's main constraint: anything the agent does that does not go through the gateway is invisible to the trainer. Tool calls, file edits and sandbox state have to be represented in the reward function and the trajectory structure, not captured automatically.
Two ways in: the rllm CLI or the decorator API
The CLI path needs no Python. The README gives three commands: rllm model setup to configure a provider, rllm eval gsm8k to evaluate, and rllm train gsm8k to train. The README also states that rllm eval <name> auto-pulls and runs a benchmark, and that 60+ benchmarks are integrated, listing Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500 and GPQA among them. The Python path is two decorators and a trainer. You write @rllm.rollout on a function taking (task: Task, config: AgentConfig) and returning an Episode, and @rllm.evaluator on a function taking (task, episode) and returning an EvalOutput with a reward, an is_correct flag and a list of Signal objects. You then construct AgentTrainer(backend="tinker", agent_flow=solve, evaluator=score, config=config, train_dataset=dataset) and call trainer.train(). Installation is explicit about which backend you get by default: uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git" installs the tinker backend only, while rllm[verl] and rllm[fireworks] are separate extras. Python >= 3.11 is required. Note that the README's PyPI badge line is commented out, so the git URL is the documented install path rather than a plain package name.
Where the abstractions leak: reward design and non-Python agents
The evaluator decorator returns a scalar reward, and the example is exact string matching against task["ground_truth"]. That is fine for GSM8K and MATH-500. It is a poor fit for the agentic benchmarks rLLM advertises, where the outcome is a patch that either passes tests or does not, or a terminal session that either completes a task or does not. The framework gives you the Signal list for partial credit, but it does not tell you how to decompose a long-horizon rollout into signals, and the README offers no guidance beyond the cookbooks pointer. The second leak is the harness claim. The README says "any harness" and lists 10+ CLI harnesses, but a CLI harness is a process, and the gateway captures HTTP calls, not subprocess behaviour. How a Claude Code or Codex process is made to route its requests through config.base_url, and what happens when that process spawns its own subprocesses, is not described in the material available. Treat the harness list as a claim to check against the documentation site rather than as a verified property. Third, the decorator API is Python-only. If your agent is a Node service or a compiled binary, the @rllm.rollout path does not apply to you.
Backend choice is a real fork, not a flag you can ignore
The README describes three training backends behind one API: verl for distributed multi-GPU, tinker for single-machine, and fireworks for the Fireworks platform. It says you switch with one flag, and in the AgentTrainer constructor that flag is backend="tinker". The practical difference is large. Tinker is the default install and runs on one machine against the Tinker API, so it is the right starting point for a small model and a short loop. Verl is the path to multi-GPU distributed training with vLLM or SGLang, and it is a separate install extra. Fireworks is a hosted platform, which means your rollouts and gradients leave your machine. The README does not give guidance on when tinker stops being sufficient, and it does not state whether a config written for one backend transfers unchanged to another. If your plan is to prototype on tinker and scale on verl, that transition is the thing to validate early, because it is where an unstated incompatibility would cost the most time.
Compared with building directly on verl
Verl is the closest real alternative, and rLLM lists it as a backend rather than a competitor, which tells you something about the intended split. If you build directly on verl you get the distributed training machinery without an intermediate gateway, and you write your rollout as a verl-compatible function. You control the data path end to end and there is no proxy between your agent and the model server. What you give up is the reuse property: the same agent code no longer runs unchanged under rllm eval. rLLM's contribution is precisely that indirection, and it is worth the cost only if you are actually running both evaluation and training against the same agent. If you only ever train, the gateway is overhead. The other alternative is to skip RL frameworks entirely and use an agent evaluation harness, but that gives you no policy update at all, so it is not a substitute for the training half.
Maintenance, release cadence and the Apache-2.0 terms
The release history shows v0.2.1 in December 2025, a post-release two weeks later, and v0.3.0-pre in April 2026, with the last push to main in September 2026. The most recent release is a pre-release, which means the newest functionality is not a stable tag. The README is explicit that the framework depends on external moving parts: the training backends (verl, tinker, fireworks), the model providers configured through rllm model setup, and the sandbox providers (Docker, Daytona, Modal, local). Any of those changing their API is an upgrade cost that lands on you, and the framework cannot insulate you from it. The licence is Apache-2.0, which permits commercial use and modification and requires that you retain the licence and attribution notices; it also includes an explicit patent grant. It does not oblige you to publish your modifications. That is a general description of the licence text, not legal advice, and if you are shipping rLLM inside a product you should read the LICENSE file and the NOTICE requirements yourself. There is no stated support commitment or long-term release policy in the material.
Editorial conclusion
Adopt rLLM if you already have a Python agent and want to reuse the same code for evaluation and RL, and if you are willing to install the extra for the backend you intend to use (rllm[verl] for multi-GPU, rllm[fireworks] for the hosted platform). Do not adopt it if your agent lives in a non-Python runtime, or if you need a documented answer on how the model gateway behaves under a real distributed training job before you commit. Verify first: whether the tinker backend alone can reach the throughput your training run needs, and whether the harnesses you actually use are covered by the 10+ CLI list or only by the Harbor-compatible task directory path.
Community notes