TextRL: a thin TRL wrapper for GRPO, DPO and KTO training

What TextRL adds on top of TRL

TextRL is not a training framework. It is a wrapper around HuggingFace TRL that reduces the setup work for text-generation RL: a single TextRLConfig dataclass, one trainer class per algorithm family, reward functions written as ordinary Python callables, and support for PEFT, accelerate and vLLM that the README describes as first-class. The intended reader is someone who has already chosen TRL and finds the per-algorithm boilerplate repetitive. If you have not chosen TRL, TextRL does not help you choose; it assumes that decision.

The algorithm table is the real content of the project. Online methods cover GRPO, RLOO and REINFORCE++. Pairwise preference covers DPO, IPO, Hinge, APO in zero and down variants, BCO-pair, NCA-pair, Robust-DPO, AOT, DiscoPOP, SPPO-hard and EXO-pair, all routed through one DPOTrainer with a unified loss_type. KTO handles binary feedback, and RewardModelTrainer handles pairwise reward training. That is a wide surface for a package whose install line is one dependency chain.

The v1.0 release is a break, not an addition. The README states plainly that the legacy PFRL/gym API (TextRLEnv, TextRLActor, train_agent_with_evaluation) is gone and points to docs/migration.md. Anyone arriving from an older tutorial that imports TextRLEnv will get an ImportError, not a deprecation warning.

How the trainers and reward callables fit together

The data flow is TRL's, with a thinner front. You load a policy, tokenizer and optional reference model through load_model, build a dataset with a helper from textrl.data, declare a TextRLConfig, and hand all of it to OnlineTrainer, PreferenceTrainer or RewardModelTrainer. The trainer then delegates to the matching TRL trainer named in the README's table. TextRL adds no scaffolding of its own for distributed runs; the README says to launch through accelerate and notes that TextRLConfig.distributed is forwarded to TRL through the extra field.

Rewards are where the wrapper earns its place. A reward is a plain callable with the signature TRL expects, taking prompts, completions and keyword columns and returning a list of floats. The @reward_fn decorator coerces it into a RewardFn protocol object. For stateful rewards such as a loaded classifier, you subclass BaseReward instead. compose(*fns, weights=...) merges several rewards, and ClassifierReward wraps any HuggingFace pipeline, with the README showing a sentiment pipeline and a target_label argument.

Memory behaviour depends on which of these you pick. The README states that when peft is set, ref_model is None because TRL disables adapters for the reference forward pass. It also notes load_ref=False for GRPO and RLOO to save memory. Those two notes matter more than the algorithm list for anyone fitting a run onto one GPU.

Installing TextRL and running a first GRPO job

Installation is a single pip command. The core install pulls trl, transformers, peft, accelerate, datasets, torch and pyyaml, according to setup.py. Optional extras add bitsandbytes for QLoRA, vLLM for rollout, and evaluate, rouge-score and sacrebleu for the rewards extra.

pip install textrl
pip install 'textrl[quant]'
pip install 'textrl[vllm]'

A first real run is the README's GRPO quickstart. It defines a reward that scores completions by distance from 64 characters, loads a small Qwen model with a LoRA adapter, and trains on a repeated two-prompt list. The point of the example is the shape of the API, not the reward itself.

from textrl import OnlineTrainer, TextRLConfig, load_model, reward_fn
from textrl.data import from_list

@reward_fn
def length_reward(prompts, completions, **_):
    return [-abs(len(c) - 64) / 64 for c in completions]

model, tok, _ = load_model("Qwen/Qwen2.5-0.5B", peft={"type": "lora", "r": 16})

cfg = TextRLConfig(algo="grpo", output_dir="out/grpo", num_generations=8,
                   beta=0.04, learning_rate=5e-6, bf16=True)

trainer = OnlineTrainer(
    model=model, tokenizer=tok, reward=length_reward,
    train_dataset=from_list(["Write a short poem.", "Explain gradient descent."] * 32),
    config=cfg,
)
trainer.train()

Expect checkpoints under out/grpo. The alternative entry point is the CLI, which reads a YAML file. The README gives this example, though the dataset block is cut off in the rendered file, so treat the dataset key as undocumented until you check example/grpo_config.yaml in the repository.

algo: grpo
output_dir: out/grpo
learning_rate: 5e-6
num_train_epochs: 1
num_generations: 8
beta: 0.04
bf16: true

model:
  name: Qwen/Qwen2.5-0.5B

accelerate launch -m textrl.cli train --config configs/grpo.yaml
textrl-train --config cfg.yaml

Four console scripts ship with the package: textrl-train, textrl-merge, textrl-eval and textrl-dump. The README marks textrl-dump as a deprecated alias for textrl-merge, so new scripts should call textrl-merge.

What TextRL will not train, and where it stops being the right tool

The clearest limitation is in the README itself. PPO, OnlineDPO, ORPO, CPO, SimPO and binary BCO were removed in TRL 0.29 or later and are therefore not supported. TextRL does not silently fall back; it raises with a migration hint. If your existing pipeline is PPO-based, this library is the wrong tool and the error message is the confirmation.

A second constraint is the dependency floor. setup.py pins trl>=0.12.0, transformers>=4.45.0, peft>=0.13.0, accelerate>=1.0.0, datasets>=2.21.0 and torch>=2.3.0. Those are lower bounds, not tested ranges. Because the unsupported-algorithm list is defined by what TRL 0.29+ removed, a resolver that lands on an older TRL would not match the README's behaviour, and the README does not state an upper bound. Pin TRL explicitly in your own environment.

The vLLM path is narrower than it looks. The README titles that section vLLM rollout (GRPO only), so RLOO and REINFORCE++ runs cannot use it. The vLLM extra itself requires vllm>=0.6.0, and the README does not document which vLLM versions have been exercised. The example sets vllm_gpu_memory_utilization to 0.6 through the extra dict, which is a starting point rather than a tuned value.

Finally, the wrapper inherits TRL's constraints without softening them. Distributed training is accelerate's job, not TextRL's. If you need a scheduler, a custom sampler or a training loop TextRL does not expose, you are back in TRL or in raw PyTorch, and the dataclass is no longer buying you anything.

TextRL versus calling TRL directly

The honest alternative is TRL without TextRL. Every algorithm in the README's table maps to a TRL trainer by name, so the difference is not capability. It is configuration surface. TRL exposes per-trainer argument classes and expects you to construct them; TextRL collapses that into one TextRLConfig with an algo string and forwards the rest through extra. For a team running several algorithm families against the same data, that is a real reduction in glue code. For a team running exactly one DPO job, it is an extra dependency between you and the library that actually does the work.

The second alternative is writing the reward loop yourself against transformers and a policy-gradient implementation. That is more work and more room for subtle error, but it removes the TRL version constraint entirely. TextRL's value proposition depends on TRL being the right substrate for you; if it is not, the wrapper adds a layer rather than removing one.

The reward abstractions are the part least duplicated elsewhere. compose with weights, ClassifierReward over a transformers pipeline, and BaseReward for stateful scorers are small conveniences, but they are the pieces you would otherwise rewrite per project.

Maintenance signals and the licence mismatch

The repository is not archived. The last push was on 2026-04-23, which is recent enough that the project is not abandoned, and setup.py declares Development Status 5 - Production/Stable with Python 3.10, 3.11 and 3.12 classifiers. There are no retrieved releases, so the version you get from PyPI is whatever the index holds; setup.py names 1.0.0.

Upgrade cost is dominated by TRL, not by TextRL. Because the supported algorithm set is defined by what TRL 0.29+ removed, a TRL major bump can change which algos raise. The v1.0 break already did this once for the PFRL/gym API, and docs/migration.md is the only path the README offers for code written before it. Budget for reading that document if you are upgrading an existing TextRL installation rather than starting fresh.

The licence metadata is inconsistent and worth checking before you rely on it. The repository states MIT. The LICENSE file is present at the top level. setup.py, however, carries the classifier License :: OSI Approved :: Apache Software License and sets license="Apache". The README's badge links to PyPI and says nothing about licensing. This is not legal advice, but the two declarations do not agree, and anyone shipping a product on top of TextRL should read the LICENSE file and, if the discrepancy matters, ask the maintainer which one governs.

Editorial conclusion

Adopt TextRL if you already accept TRL as your training stack and want its algorithm surface behind one dataclass, a YAML CLI and composable reward callables; the v1.0 release deleted the PFRL/gym API, so anything written against TextRLEnv or TextRLActor needs the migration document before it runs again. Skip it if you need PPO, OnlineDPO, ORPO, CPO, SimPO or binary BCO, because the README states TRL 0.29+ removed them and TextRL raises a migration hint instead of training. Before committing, verify three things: which vLLM version your GRPO run resolves to, whether the installed package reports textrl 1.0.0 from setup.py or the Apache licence its classifier names, and what the README's own truncation hides about the dataset block in the YAML example.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. voidful/TextRL on GitHub