agentic-rl-lab: reproducing GRPO, DAPO, GSPO and Search-R1 on CPU-only machines
Reproducing and studying RL algorithms for LLM agents, including GRPO, GSPO, DAPO, OPD, Search-R1, ReTool, ALFWorld and beyond.
At a glance
- What is it?
- KMnO4-zx/agentic-rl-lab is a tutorial repository that reimplements a dozen LLM agent RL algorithms against a remote PyTRIO training service, so the local install stays small. The trade-off is that nothing trains locally and the whole stack depends on a hosted API.
- Who is it for?
- Adopt it if you want to read and rerun small LLM agent RL experiments without owning GPUs, and if a PyTRIO account is acceptable. Skip it if you need a self-contained trainer, a stable library API, or a pip-installable package: this is a chapter-by-chapter lab notebook, and the pyproject.toml pins pytrio==0.2.9 exactly.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What agentic-rl-lab actually is, and who should read it
The README describes the repository as an experiment log and tutorial more than a framework: it reproduces a set of LLM and agent RL algorithms and explains each one first, then ships runnable code for data, reward, loss, training loop and SwanLab logging. The chapter list maps to papers and methods rather than to modules: loss functions, GRPO, OPD in a general and a medical variant, Search-R1, OPSD, ReTool, DAPO, GSPO, ALFWorld, Vision GRPO, TEMPO, AgentOPSD and Spec-o3. That is a reading list with executable companions, and the intended reader is someone who wants to study agentic RL algorithms but has been blocked by the usual obstacles. The README names them directly: no GPUs, monolithic training-and-inference code, and the coupling in Verl's engineering code. The author's stated reason for choosing PyTRIO is exactly that reduction in setup friction, and the repository claims the whole set was studied and reproduced in under a month.
The audience is narrow in a useful way. If you are an algorithm engineer or researcher who wants to compare how importance sampling, PPO and CISPO losses differ in practice, or how DAPO's dynamic sampling behaves, the chapters give you a runnable target. If you want a library to import into production training, this is not it. There is no released package, no versioned API, and no release history in the repository metadata.
The split architecture: local data and tensors, remote PyTRIO training
The most consequential design decision is in pyproject.toml, not in the README. Under tool.uv.sources there is a comment stating that the project only does data and tensor processing locally, while training and sampling run on a remote PyTRIO service. Both torch and torchvision are pinned to the pytorch-cpu index, declared as an explicit index pointing at download.pytorch.org/whl/cpu.
So the data flow is: a chapter script prepares a dataset and builds tensors on your machine, the PyTRIO client sends the work to a hosted service where the model is sampled and the policy is updated, and results come back for logging. SwanLab handles experiment tracking. The deepseek-search dependency is also pinned, to a specific git revision rather than a version range, which keeps the Search-R1 search backend reproducible at the cost of never picking up upstream fixes without an edit.
This is the same trade-off Tinker-style services make, and the README is explicit that the author considers this class of product to be the direction for post-training infrastructure. The honest reading: you are trading control of the training loop for a much smaller local footprint. You cannot inspect or modify the sampler, and you cannot run offline.
Installing agentic-rl-lab with uv and running a first chapter
The README gives a clone-and-sync sequence for the whole repository. Python 3.13 or newer is required by pyproject.toml, and uv is the package manager used throughout.
git clone https://github.com/KMnO4-zx/agentic-rl-lab.git
cd agentic-rl-lab
uv syncAfter uv sync finishes you have the base dependencies: datasets, math-verify, matplotlib, modelscope, numpy, openai, python-dotenv, pytrio 0.2.9, swanlab 0.9.2, the CPU builds of torch and torchvision, and tqdm. Nothing heavy is downloaded from the CUDA wheels, which is the point.
Two chapters need more. ALFWorld, TEMPO and AgentOPSD require the TextWorld environment, installed through an optional extra:
uv sync --extra alfworldThat extra resolves to alfworld==0.4.2 and spacy==3.8.13. If you would rather lift a single demo script into an existing project instead of syncing the whole repo, the README lists the base dependency set as a uv add command, which is the only place the version floors are stated in that form.
Search-R1 needs one more package, pinned to a commit hash:
uv add "deepseek-search @ git+https://github.com/KMnO4-zx/deepseek-search.git@6215c8dbb7347f94e9dcea6e741df5918449d6c4"For the Spec-o3 chapter, the README points to a separate start.md rather than repeating the steps, and says it covers data download, SFT, GRPO, evaluation and plotting. That file is where you should look before running anything in 09-spec-o3, because the README does not restate its commands. What you should expect to see after a run is a SwanLab experiment with the recorded metrics for that chapter; the README does not document a checkpoint layout or an output directory convention.
Where agentic-rl-lab breaks down
The dependency on a hosted service is the first failure mode. The README's own framing is that PyTRIO removed the barrier of not having GPUs. Remove PyTRIO and there is no local path to run these algorithms: pyproject.toml deliberately installs CPU-only torch, so the repository as shipped cannot train on your own hardware even if you have it. If your requirement is an air-gapped or fully self-hosted setup, this repository does not provide one.
Access is also gated in a way that is easy to miss. The README mentions that you can email a proposal to apply for PyTRIO vouchers, and that the review is strict. That implies paid or credit-based access to the service; the repository does not state pricing, quotas or what happens to in-flight runs when credits run out.
The version pinning is a second constraint. pytrio==0.2.9 is an exact pin, and deepseek-search is pinned to a commit. When the service API moves, these chapters stop working until someone updates the pin, and there is no release history in the repository metadata to signal whether that maintenance happens on a schedule. The last push to main was on 2026-09-17, so the repository is currently being touched, but a recent push is not a compatibility guarantee.
Finally, the README does not document rollback, resume-from-checkpoint, or how to reproduce a specific run's results after the fact. For a tutorial that is acceptable. For anything you intend to cite in a paper, it is a gap you will have to close yourself.
Compared with Verl and other agentic RL codebases
The README positions the project against Verl directly, saying the author was put off by Verl's highly coupled engineering code. The difference is not just size, it is what each one optimizes for. Verl is a training framework: you install it, point it at a cluster, and it manages distributed rollout and update at scale. agentic-rl-lab is a set of chapter directories, each with its own readme, data preparation and training script, backed by a remote service that owns the sampler and the optimizer.
That means the two answer different questions. If you need to train a model on your own GPUs with your own parallelism strategy, Verl is the shape of tool you want and this repository will frustrate you immediately, because the CPU-only torch index makes local training impossible. If you need to understand why GSPO moves the importance ratio and clipping from the token level to the sequence level, or how ReTool interleaves code execution with a local sandbox, a chapter with a runnable script and a written explanation gets you there faster than reading a framework's source tree.
Other alternatives in the same space are the survey literature and the awesome-list style collections that the related searches point at. Those give you the map of agentic RL methods without any executable code. agentic-rl-lab sits between the two: less complete than a survey, far more runnable than a paper list.
Licence, maintenance and the cost of keeping chapters working
The repository is Apache-2.0 and the README defers to the LICENSE file. For the code in the repository that is a permissive arrangement: you can reuse the chapter scripts in your own work, subject to the usual attribution and notice conditions, and the file itself is the authority on the exact terms. The dependencies are separate. alfworld, spacy, pytrio, swanlab, deepseek-search and the rest each carry their own licence, and the deepseek-search package is pulled from a personal fork at a fixed revision, which is worth checking before you redistribute anything built on it.
Maintenance cost is dominated by the pins. Upgrading pytrio means re-verifying every chapter, because the loss implementations in 00-loss-function and the training loops in later chapters are written against that exact client. Upgrading alfworld or spacy touches three chapters at once. The deepseek-search revision can only move by editing pyproject.toml and re-running Search-R1, which is the cheapest of the three to test.
There is no changelog and no release history, so the practical way to judge whether a chapter still works is to run it and watch SwanLab. The pyproject.toml comment about remote training is the single most useful line for predicting what will break first: any change to the PyTRIO service API invalidates the client calls in every chapter at once.
Editorial conclusion
Adopt it if you want to read and rerun small LLM agent RL experiments without owning GPUs, and if a PyTRIO account is acceptable. Skip it if you need a self-contained trainer, a stable library API, or a pip-installable package: this is a chapter-by-chapter lab notebook, and the pyproject.toml pins pytrio==0.2.9 exactly. Before committing, verify that your PyTRIO credits cover a full run of the chapter you care about, and check whether the 09-spec-o3 start.md still matches the code in that directory.
Frequently asked questions
What is RL in agentic AI?
In this repository the term covers reinforcement learning applied to LLM agents: the model samples trajectories, a reward or a teacher scores them, and a policy loss such as importance sampling, PPO, CISPO, GRPO, DAPO or GSPO updates the model. The chapters walk through each of those algorithms with runnable code and a written explanation.
What is an agent in RL?
The repository's agent chapters treat the agent as a model that takes multiple turns and calls tools: Search-R1 does multi-turn search, ReTool interleaves code with a local sandbox, and ALFWorld trains a household agent against a real TextWorld environment. The reward comes from the task outcome rather than from a single answer.
Is RL a dead end?
The repository does not argue that position either way. It is a reproduction and study repository that assumes the algorithms are worth understanding, and it covers methods from GRPO through DAPO, GSPO and TEMPO without making a claim about the field's direction.
What is reinforcement learning (RL)?
The repository does not define the general concept. Its 00-loss-function chapter starts from the losses instead, explaining what importance_sampling, ppo and cispo each optimize, which assumes you already know the basic RL loop.
Community notes