prime-rl: asynchronous RL training for agentic models at 1000+ GPU scale
Project brief: Agentic RL Training at Scale. It is designed to be easy to use and hackable, yet capable of scaling to 1000+ GPUs.
At a glance
- What is it?
- prime-rl is a Python framework for post-training large language models with asynchronous reinforcement learning. It pairs an FSDP2 trainer with vLLM inference and the verifiers environment stack, and it needs at least one NVIDIA GPU before anything runs.
- Who is it for?
- prime-rl is for teams that already have NVIDIA GPUs and want asynchronous RL on agentic environments rather than a synchronous PPO loop. It is not for anyone without a GPU, and not for a first experiment in RL fine-tuning, because the setup path assumes you can debug a multi-process stack.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 18, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What prime-rl solves, and who it is aimed at
Reinforcement learning on language models stops being a single-GPU exercise the moment the policy has to generate long agentic trajectories. Generation is slow, training wants the GPU back, and a synchronous loop leaves the trainer idle while the sampler works. prime-rl is built around that split. The README describes it as "Fully asynchronous RL for high-throughput agentic training at scale", with the trainer and the inference engine as separate processes that do not have to take turns.
The intended user is a research or infrastructure engineer who already runs multi-node jobs. The README states the project is designed to scale to 1000+ GPUs and to train 1T+ MoE models, and it lists Slurm and Kubernetes as deployment targets. That is a different audience from someone fine-tuning a 7B model on a single consumer card, even though the project does support small runs: the debug configs are explicitly marked as requiring one GPU.
The second audience is environment authors. prime-rl integrates with verifiers environments through the Environments Hub, and the README calls out built-in support for SWE and agentic environments. If your work is writing reward-bearing environments rather than tuning kernels, the framework is meant to consume what you produce.
The asynchronous loop: orchestrator, inference server, trainer
The architecture is visible in the console scripts declared in pyproject.toml. There is no single monolithic entry point for a full run; instead there are separate commands for `rl`, `sft`, `inference`, `trainer`, `orchestrator`, `evals`, `env-server` and `dashboard`. A full RL job composes several of them.
The inference server is vLLM. The trainer is FSDP2. The orchestrator sits between them, and the README's validation step for the full stack names all three: "inference + orchestrator + trainer". The orchestrator is what makes the loop asynchronous. It drives environment rollouts against the inference server and feeds the resulting trajectories to the trainer, rather than blocking on a generation phase inside the training step.
On the training side, prime-rl ships optimized model code under `src/prime_rl/trainer/models/`. The README describes expert parallelism for MoE layers and context parallelism for long sequences, and notes that with the default `[model] impl = "auto"`, the trainer picks that custom stack when the Hugging Face config type is registered. For unregistered architectures it falls back to `impl = "hf"`, and the support table shows that fallback losing EP and, for some families, CP as well. That is the trade-off to understand before picking a model: the fast path is a registry, and being outside it costs you parallelism options.
Installing prime-rl and running the two-GPU end-to-end check
The README gives a one-line installer, which is the fastest path to a working checkout:
curl -sSL https://raw.githubusercontent.com/PrimeIntellect-ai/prime-rl/main/scripts/install.sh | bashThe manual path is more informative if you want to know what landed on disk. It clones the repository, initializes four submodules under `deps/`, installs uv, then syncs the lock file:
git clone https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl
git submodule update --init -- deps/verifiers deps/renderers deps/prime-envs deps/pydantic-config
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv sync --all-extrasOne detail in the README is easy to miss and will cost you time: environments are opt-in uv workspace members, so `uv sync --all-extras` does not install them. To train on environments you need `uv sync --all-extras --all-packages`, or a narrower `uv sync --package prime-rl --package <env>`.
The README's validation sequence is the honest way to find out whether the install worked, because it tests each process separately before asking them to cooperate. Python 3.12 is required, `requires-python = "~=3.12.0"` in pyproject.toml:
uv run python -V
uv run python -c "import flash_attn"
uv run trainer @ configs/debug/fake/rl.toml
uv run inference --vllm.model Qwen/Qwen3-0.6BThe first two commands should print a 3.12 version and exit silently. The trainer and inference checks each need one GPU. The end-to-end check needs two, and is the one that actually exercises the orchestrator:
uv run rl @ configs/basic/reverse-text/rl.tomlIf you want run logging, the README points at `uv run wandb login` or `export WANDB_API_KEY=...`, and for gated Hugging Face assets at `uv run hf auth login` or `export HF_TOKEN=...`. The opt-in Flash Attention 3 build is Hopper-only and compiles from source; the README warns that afterwards you cannot run plain `uv sync --all-extras` or `uv run` without uninstalling it, and suggests `uv sync --inexact` or `uv run --no-sync` instead.
The GPU requirement and other places prime-rl is the wrong tool
The README is blunt about the hardware floor: "Currently, you need at least one NVIDIA GPU to use prime-rl." That single sentence rules out a large class of potential users. There is no CPU training path documented, no Apple Silicon path, and no AMD path in the prerequisites. If you are evaluating frameworks on a laptop before requesting cluster time, prime-rl will not get you past the install validation.
The second limitation is the model registry. The support table is long but finite, and it is organized by Hugging Face config type. Families listed with EP and CP include GLM-5, Qwen3 MoE, Qwen3.5 MoE, Poolside Laguna, MiniMax M2, Nemotron H, Trinity, GLM-4 and GPT-OSS MoE. The VLMs are the exception: the table shows multimodal training for Qwen3 and Qwen3.5 VLMs with EP and CP marked only for the MoE variants, and CP marked as unavailable for the VLM row. A dense Hugging Face causal LM outside the registry gets `impl = "hf"`, which the table shows without EP.
The third limitation is operational. This is a multi-process system with a trainer, an inference server, an orchestrator and an environment server, and the debugging surface scales with that. The README asks users to file an issue when a setup fails, which is a fair signal that not every configuration is smooth. If your problem is a single-node SFT run on a small dense model, the asynchronous machinery buys you nothing and only adds processes that can fail independently.
How prime-rl differs from verl
verl is the comparison people reach for, and the difference is in where the asynchrony lives. verl's published design centers on a hybrid controller that coordinates a training worker group and a rollout worker group, with the synchronous mode as the default and asynchronous rollout as an option layered on top. prime-rl makes asynchrony the premise of the architecture rather than a mode: the README's first differentiator is "Fully asynchronous RL", and the process split into separate `inference`, `orchestrator` and `trainer` entry points reflects that.
The second difference is the environment layer. prime-rl depends on `verifiers[harbor]` and pulls `deps/verifiers`, `deps/renderers` and `deps/prime-envs` as submodules, and it integrates with the Environments Hub. That makes the environment a first-class artifact you install as a uv workspace package. In verl, environment and reward logic tends to be code you write inside your training script. Neither is better in the abstract; if you already have a verifiers environment, prime-rl removes a layer of glue, and if you have reward code written against another interface, prime-rl asks you to port it.
The third difference is the model stack. prime-rl ships its own optimized `ModelForCausalLM` implementations with EP and CP and a `quack-kernels` dependency. That buys throughput on the listed MoE families and costs you the ability to drop in an arbitrary architecture and expect the same parallelism.
Licence and the cost of tracking releases
prime-rl is Apache-2.0. That is a permissive licence, and it matters here because the project pulls in a large dependency set whose terms are not all the same: vLLM, torch, flash-attn, mooncake-transfer-engine and ring-flash-attn are separate projects with their own licences, and the submodules under `deps/` are separate repositories. Apache-2.0 on prime-rl says nothing about those. Check the dependency licences you actually ship, particularly if you redistribute a container built from `Dockerfile.cuda`.
Upgrade cost is real. The release cadence visible in the repository is roughly one minor version per month: v0.7.0 on 2026-07-14, v0.8.0 on 2026-08-07, v0.9.0 on 2026-08-25. The last push to the default branch was on 2026-08-25. Dependencies are pinned tightly in places, including `transformers==5.6.2`, so a version bump can move several packages at once. The submodule pins mean an upgrade also moves `deps/verifiers` and `deps/renderers`, which is where environment behaviour changes would surface. If you pin a working combination, expect to re-run the full validation sequence, not just the trainer check, when you move it.
Editorial conclusion
prime-rl is for teams that already have NVIDIA GPUs and want asynchronous RL on agentic environments rather than a synchronous PPO loop. It is not for anyone without a GPU, and not for a first experiment in RL fine-tuning, because the setup path assumes you can debug a multi-process stack. Before committing, verify that your model family is listed in the support table, that `uv run rl @ configs/basic/reverse-text/rl.toml` completes on two GPUs, and that the environment you intend to train on is installed as an opt-in workspace package rather than pulled in by `uv sync --all-extras` alone.
Frequently asked questions
What is prime-rl?
It is a Python framework for large-scale reinforcement learning on language models, described in its README as fully asynchronous RL for agentic training at scale. It uses FSDP2 for training and vLLM for inference, and it integrates with verifiers environments through the Environments Hub.
How does prime-rl compare with verl?
prime-rl makes asynchrony the architecture, with separate inference, orchestrator and trainer entry points, while verl's published design uses a hybrid controller with synchronous rollout as the default. prime-rl also treats verifiers environments as installable uv workspace packages, where verl-style setups usually keep reward logic inside the training script.
What are the prerequisites for installing prime-rl?
The README states you need at least one NVIDIA GPU, and the project requires Python 3.12. Development and testing are listed on NVIDIA RTX 3090, 4090, 5090, A100, H100, H200 and B200 hardware.
Which models does the prime-rl trainer support?
The support table covers GLM-5, Qwen3 MoE, Qwen3.5 MoE, Poolside Laguna, MiniMax M2, Nemotron H, Trinity, GLM-4 and GPT-OSS MoE families with expert and context parallelism. Other Hugging Face causal LMs run through `impl = "hf"` without expert parallelism.
Does prime-rl install verifiers environments automatically?
No. The README notes that environments are opt-in uv workspace members, so `uv sync --all-extras` does not install them. You need `uv sync --all-extras --all-packages`, or `uv sync --package prime-rl --package <env>` for a subset.
Community notes