QeRL: NVFP4 Quantization Plus LoRA for RL Training of 32B LLMs on One H100

The memory wall QeRL is built to move

Reinforcement learning for reasoning models is expensive in a way supervised fine-tuning is not. Every training step needs rollouts: the policy generates full reasoning traces, a reward function scores them, and the gradients flow back. Long traces mean long generations, and generation is where the GPU hours go. On top of that, the standard recipe keeps a policy model, a reference model and an optimizer state resident at once. The README states the problem directly: RL "is resource-intensive, requiring substantial GPU memory and long rollout durations." A 32B model in 16-bit weights alone is roughly 64 GB before you add a second copy or any optimizer state, which puts it past a single 80GB card. QeRL's target user is the researcher or small team that has one H100 and wants to run GRPO-style training on a model that size. The README frames the payoff as a single sentence: RL for 32B LLMs on a single H100 80GB GPU, with full-parameter fine-tuning performance maintained. That is a memory claim first and a speed claim second, and the two are worth separating when you evaluate it.

How NVFP4 weights, LoRA adapters and adaptive noise fit together

The mechanism has three parts. First, the base model weights are quantized to NVFP4, a 4-bit floating point format, using the llm-compressor tooling bundled in the repository. The README describes QeRL as using "NVFP4 for low-cost and fast LoRA training with injected adaptive quantization noise." Second, training does not update those weights. LoRA adapters are trained instead, which is what keeps the optimizer state small enough to fit alongside the quantized base. Third, and this is the part that departs from a standard efficiency story, QeRL injects quantization noise during training and adjusts it with a mechanism the paper calls Adaptive Quantization Noise, or AQN. The README's argument is that the noise is not purely a cost. Quantization noise raises policy entropy, higher entropy encourages exploration, and better exploration produces faster reward growth. Two of the README figures make this claim visually: one showing higher initialized entropy from quantization noise, another showing a quantized model converging to a better reward score under GRPO. Treat that as the paper's thesis rather than a settled result. The repository gives you the code to reproduce it, not an independent verification.

Installation is a two-environment job, not one

There is no pip install qerl. The README splits the work across two conda environments. The first handles training. It requires an NVIDIA GPU that supports the NVFP4 weight format with triton (the README names RTX 5090, H100 and B100 as examples), a Linux operating system, and 64 GB of RAM. The commands are: git clone https://github.com/NVlabs/QeRL, then cd QeRL, then conda create -n qerl python=3.10 -y, then conda activate qerl, then conda install nvidia/label/cuda-12.4.1::cuda, then conda install -c nvidia/label/cuda-12.4.1 cudatoolkit, and finally sh setup_env.sh. The second environment builds the quantizer. You cd into llm-compressor, create a python=3.12 environment, pip install -e . and pip install nvidia-ml-py, then run quantize_nvfp4.py per model. The README lists four invocations covering Qwen2.5-3B-Instruct, 7B, 14B and 32B. Note the Python version split: 3.10 for training, 3.12 for quantization. If you try to collapse this into one environment you are off the tested path, and the README says as much about hardware: other setups "could also work but hasn't been tested."

What a training run actually looks like

Training is driven by shell scripts rather than a CLI. The README gives two entry points. bash training/dapo_qwen2.5-7b_nvfp4_single_gpu.sh runs QeRL, and bash training/dapo_qwen2.5-7b_bf16_single_gpu.sh runs vanilla 16-bit LoRA for comparison. The naming tells you the algorithm family: DAPO, run on Qwen2.5-7B, single GPU, in two precision modes. That pairing is useful because it gives you a controlled baseline without writing one. The one configuration key the README documents explicitly is --vllm-gpu-memory-utilization. It controls how much of each GPU vLLM is allowed to claim, and the README's advice is counterintuitive for anyone coming from standard inference serving: set it smaller than usual for QeRL. The reason is that quantization already cut the weight footprint, so the rollout engine does not need its customary share, and the freed memory belongs to the training side. That single flag is where most of the practical tuning lives, and the README treats it as the main knob rather than one of several.

The rollout speedup claim and what it depends on

The README states a rollout-phase speedup of over 1.5x, and adds a qualifier worth reading closely: the gain grows with longer reasoning traces. That qualifier matters more than the number. Rollout cost scales with generated tokens, so a method that speeds up decoding pays off proportionally to how much the model writes. On a short-answer task the advantage shrinks toward the fixed overhead of the training loop; on a long chain-of-thought task it compounds. The README also includes a figure titled "Rollout throughput of Qwen2.5-14B/32B-Instruct models under different lora ranks," which implies rank is a second lever on throughput. The repository does not state a recommended rank in the material available here, so treat rank selection as something you determine by measurement on your own workload rather than by copying a default. The 1.5x figure is a paper claim, measured on the authors' setup. Your hardware, your sequence lengths and your LoRA rank will move it.

Where QeRL is the wrong tool

The hardware requirement is the hardest constraint. NVFP4 support with triton is not universal, and the README's tested list is recent data-center and consumer parts. If you are on A100 or an older accelerator, this repository is not for you, and no configuration flag changes that. The second limitation is the LoRA restriction itself. QeRL trains adapters, not base weights. The README reports that this matches full-parameter fine-tuning on GSM8K (90.8%) and MATH 500 (77.4%) at the 7B scale, which is a meaningful result on those benchmarks. It is not a general guarantee. If your task needs the base weights to move, or if you are studying full-parameter RL dynamics as the object of research, LoRA is the wrong substrate regardless of how well it scores on math. Third, there are no tagged releases in the material retrieved. You are tracking the main branch of a research repository, and the last push is recent. There is no version to pin to, which makes reproducibility a matter of recording the commit hash yourself.

How it differs from QLoRA and from a serving-layer quantizer

The obvious comparison is QLoRA, and the README makes it directly: QeRL is reported to achieve faster reward growth and higher final accuracy than both 16-bit LoRA and QLoRA. The architectural difference is where quantization sits in the loop. QLoRA quantizes the frozen base to shrink memory and then trains adapters on top; the quantization is a storage decision, applied once, and the noise it introduces is something you tolerate. QeRL treats the quantization as part of the training dynamics. The NVFP4 weights stay quantized through rollout, and the noise level is actively modulated by AQN rather than fixed at quantization time. That is the actual distinction: QLoRA optimizes for fitting, QeRL optimizes for fitting plus an exploration effect it claims is beneficial. A second comparison point is against quantizing only at inference. Tools that produce a 4-bit checkpoint for serving leave the RL loop in higher precision, which means the memory problem QeRL targets never gets solved. QeRL's bet is that the same quantization can serve both roles.

Maintenance, licence and what to check before you commit

The repository is Apache-2.0, which permits commercial use and modification, and it carries no retrieved releases, so there is no changelog to read for breaking changes. The dependency surface is the real maintenance cost here: a CUDA 12.4.1 conda channel, a separate llm-compressor environment installed with pip install -e ., and a vLLM-based rollout path whose memory behaviour you tune through --vllm-gpu-memory-utilization. Each of those moves independently of QeRL, and a CUDA or vLLM bump can invalidate the tested combination. Budget for pinning rather than upgrading on a schedule. On licence, Apache-2.0 covers the code; the quantized checkpoints you produce from Qwen2.5 weights inherit whatever terms those models carry, and this article is not legal advice on that. The concrete thing to verify first is the quantizer. Run python quantize_nvfp4.py --model Qwen/Qwen2.5-3B-Instruct end to end, load the output, and confirm it serves before you spend H100 hours on the 32B path. If the 3B checkpoint does not load cleanly, the 32B run will not either.

Editorial conclusion

QeRL is aimed at research groups and platform engineers who already have NVFP4-capable NVIDIA hardware, a Linux host with 64 GB of RAM, and a reason to run GRPO-style RL on models in the 7B to 32B range without a multi-node cluster. It is the wrong tool if your GPUs predate the NVFP4 path (the README lists RTX 5090, H100 and B100 as the tested class), if you need a stable tagged release rather than a main branch, or if your work depends on full-parameter updates for reasons LoRA cannot satisfy. Before adopting it, verify three things against your own setup: that the quantize_nvfp4.py script in llm-compressor produces a checkpoint your serving stack can load, that the training script you intend to run actually exposes the --vllm-gpu-memory-utilization flag, and that the reward function in training/dapo_qwen2.5-7b_nvfp4_single_gpu.sh matches your task rather than the math benchmarks the paper reports.

Official sources

1. Issues
2. License: Apache-2.0
3. NVlabs/QeRL on GitHub
4. README