miniVERL: running verl's PPO and GRPO on one consumer GPU by taking turns instead of scaling out

Post-training reinforcement learning without a cluster

verl is a reinforcement-learning post-training framework aimed at multi-GPU and multi-node setups. miniVERL takes the same programming surface and points it at a single consumer NVIDIA GPU. The README's one-line summary is exactly that: verl for a single consumer GPU, running PPO, GRPO and on-policy distillation locally, with support for bringing an existing verl config tree and picking up interrupted runs.

The audience is a researcher or engineer who has a verl workflow, or wants to learn one, but not a cluster to run it on. That includes students, people prototyping a reward function before renting hardware, and anyone who wants to reproduce a training run on a workstation. The project is Apache-2.0 and marks itself Development Status 4 - Beta in `pyproject.toml`, and the README states PyPI `v0.16.0` is stable while `main` is development.

The reason this is more than a wrapper is the compatibility promise. The README says you can reuse verl fields and overrides and inspect how each maps to local execution, which means it is trying to be a faithful local implementation of verl's semantics rather than a separate framework that merely resembles it.

Phase scheduling: one GPU playing every role in turn

The mechanism that makes single-GPU training fit is temporal, not spatial. The README states that the actor, critic, reference, reward and teacher roles share the GPU in phases. A cluster gives each role its own hardware and runs them concurrently. miniVERL runs them one after another on the same card, so peak memory is set by the largest single phase rather than the sum of all roles.

The algorithms differ in which roles they instantiate. PPO trains an actor and a separate critic. GRPO drops the separate critic in favour of grouped rewards, which is part of why it fits smaller memory budgets. On-policy distillation, which the README abbreviates OPD, learns from teacher targets computed on the student's own current rollouts. Each is a different composition of the same phase machinery.

The project also treats recovery as a first-class feature rather than a checkbox. The README says a run can resume the model, optimizer, data position and random state from a checkpoint, and that an interruption reproduced byte-identical trajectories, adapter and optimizer tensors on resume. On a single workstation that people reboot and reclaim for other work, exact resumption is the difference between an interrupted run being a pause and being a loss.

From install to a first trained adapter

The README separates the CUDA PyTorch build from the package, because the right PyTorch depends on your CUDA version. You install PyTorch through its official installer, then the package with the extras you need:

python -m pip install "miniverl[train,hydra]"

The first experiment is a Qwen3-0.6B example trained against a simple length reward, and the README walks it in three commands: sample data, dry-run the config, then run it for real:

miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example hydra-ppo --bind reward.provider=target_length --dry-run
miniverl run --example hydra-ppo --bind reward.provider=target_length --run-id local-ppo

The `--dry-run` step is the one to respect: it validates the configuration before committing GPU time, which on a single card is your only card. After the run, the README says to inspect `rewards.jsonl` to see the reward signal the training actually saw. An existing verl user can instead point miniVERL at their own config tree with `--verl-config-path` and pass Hydra overrides such as `algorithm.adv_estimator=grpo`, which is how the compatibility claim is meant to be exercised in practice.

The reported memory numbers, read carefully

The README publishes measured figures rather than a vague promise of thrift. On an RTX 4080, the Qwen3-0.6B/1.7B OPD workload completed eight updates over 32 distinct prompts at 3.1914 GiB peak reserved VRAM, and a smaller SmolLM2-360M/1.7B workload used 1.4961 GiB. It also records that an interruption after update four reproduced byte-identical results on resume.

Those numbers are worth reading with their scope attached. They describe specific small models, a 32-prompt workload and a particular reward profile, and the README links a dedicated workload document and a 24-cell rollout-backend study behind them. The precision to four decimal places signals the measurement is real, but it is a measurement of a tiny run. It tells you the phase-scheduling approach genuinely keeps peak memory low on small models. It does not promise that a much larger model or a longer context will fit, and the README is careful to point at a hardware-planning guide and a limitations document instead of implying it will.

The README also states, to its credit, that the linked studies include negative and mixed results with their original data and scope. A post-training project that publishes its failures alongside its wins is describing a real experimental record rather than a sales sheet.

Where the single-GPU boundary bites

The honest limitation is the premise itself. miniVERL exists to run on one GPU, and the README names a distributed-execution boundary and directs readers to a limitations document for hardware coverage and scientific scope. Sequential phase execution buys low peak memory by paying in wall-clock time: roles that a cluster runs at once, miniVERL runs in turn, so a run that fits will still take longer than the same run spread across several cards.

There is also a fit ceiling that no amount of scheduling removes. Model choice depends on GPU memory and workload, and the compatibility matrix, not the README's headline, is where the supported upstream verl versions and semantics are pinned. Reusing a verl config is not a guarantee that every field behaves identically, only that documented profiles are supported.

The right way to read this project is as a local rehearsal and teaching tool. When a workload outgrows one card, the README's own scale-out path, exporting the PEFT adapter, the Parquet data and the config with a readiness report, is the intended exit, which is an admission that the single GPU is a starting line rather than a destination for large runs.

Against renting a multi-GPU verl run in the cloud

The direct alternative is to skip local training and run verl itself on rented multi-GPU cloud hardware. That path has a higher ceiling on the first day: bigger models, real parallelism and the full framework without a single-card constraint.

What it costs is the tight iteration loop miniVERL is built for. Every cloud experiment meters money and adds provisioning between an idea and a result, which discourages the quick, throwaway runs that teach you how a reward function behaves. miniVERL keeps that loop on a workstation at no marginal cost, and its export path is designed to hand a validated small-scale setup to exactly that larger environment when the time comes. So the two are stages rather than rivals: use miniVERL to design the reward, verify resumption and get a small run correct, then take the exported config and adapter to rented hardware when the model or the workload no longer fits one card.

Licensing, PyPI cadence and upgrade cost

miniVERL is an independent Apache-2.0 project, which permits commercial use and derivatives with attribution and carries the Apache patent grant, and the repository ships a `THIRD_PARTY_NOTICES.md` acknowledging upstream code. For a project that mirrors another framework's interface, keeping that dependency picture explicit matters.

Distribution is PyPI under the name `miniverl`, with `v0.16.0` marked stable and `main` as development, so users get versioned releases while contributors track the moving branch. The install model splits responsibility deliberately: the CUDA PyTorch build is chosen separately through PyTorch's installer, and the package's extras (`train`, `hydra`, quantization) add only the Python dependencies. That split is the main upgrade hazard, since a PyTorch and CUDA mismatch is the usual way a working setup breaks, but it is also what lets one package serve many CUDA versions.

Developers install editable with the dev extras and run the test suite excluding GPU and network markers, which lets contribution and CI proceed on a machine without a GPU at all:

pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"

The deeper upgrade cost is tracking verl. Because compatibility with upstream semantics is the whole value proposition, the compatibility matrix is the file to watch on every bump, since it defines exactly which verl versions and behaviours a given miniVERL release actually honours.

Editorial conclusion

Choose miniVERL if you have a verl workflow, or want to learn one, and a single NVIDIA GPU rather than a cluster: the phase-scheduling design and byte-identical resume are what make small PPO, GRPO and OPD runs practical on a workstation. Do not choose it for large models or long contexts, where the single-GPU boundary and sequential execution make a rented multi-GPU verl run the better path. Before committing GPU time, install the CUDA PyTorch build separately, run a --dry-run of the hydra-ppo example, and read the compatibility matrix to confirm your verl config's fields are supported.

Frequently asked questions

Official sources

1. DaoyuanLi2816/mini-verl on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases