TTRL: reinforcement learning on unlabeled test data, built on verl
[NeurIPS 2025] TTRL: Test-Time Reinforcement Learning
At a glance
- What is it?
- TTRL replaces ground-truth reward with a majority-vote reward so a reasoning model can keep training at inference time. The repository is a fork of verl, and it inherits both verl's configuration surface and verl's hardware demands.
- Who is it for?
- Adopt TTRL if you have a reasoning model, a benchmark whose answers are checkable by string match, and enough GPU memory to run inference and training at the same time; the repository states that all experiments used 8 x NVIDIA A100 80GB, and the AIME 2024 reproduction is a single script. Do not adopt it if your task has no comparable answer format, since the majority-vote reward depends on answers that can be tallied.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 154 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The label problem TTRL is built around
Supervised fine-tuning and standard RLVR both need something to grade the model's output. For math and similar reasoning tasks that something is usually a ground-truth answer string. TTRL targets the case where you have the questions and no answers at all. The README states the goal plainly: online RL on data without ground-truth labels, especially test data. The intended user is a researcher or engineer who already has a reasoning checkpoint and wants to squeeze more pass@1 out of it on a specific evaluation set, without paying for labels or human annotation. That framing matters, because it means TTRL is not a general training framework. It is a procedure for a narrow situation: a fixed set of prompts, a model that can already solve some of them, and an answer format regular enough that the same answer can appear more than once across samples.
Majority voting as the reward signal
The mechanism is stated in the introduction and shown in the pseudo-code figure: common test-time scaling practices such as majority voting yield rewards good enough to drive RL. In practice the reward function is the only thing that changes relative to a normal RLVR pipeline. The model samples n completions per prompt, the answers are extracted and tallied, and the most frequent answer becomes the effective label for that prompt. Completions that agree with it get positive reward, the rest do not. The README calls the resulting metric maj@n and notes something worth pausing on: TTRL is supervised only by maj@n, yet the reported results surpass that metric as an upper bound on the initial model and approach models trained with ground-truth labels. That is the interesting claim in the paper. It also tells you where the method is fragile. If the model's samples are spread across many different answers, the majority is not a signal, it is noise, and the reward teaches the model to imitate its own confusion. There is no separate verifier in the loop.
What the repository actually contains
TTRL is a fork of verl, and the directory layout reflects that: the setup instructions tell you to cd into TTRL/verl before doing anything else, and the README points at the verl documentation for code details. The repository ships example scripts under verl/examples/ttrl for several models and benchmarks, plus a data/preprocess.py utility that converts JSON data into Parquet for verl to consume. Two release tags are listed, verl (v2.0.0) and OpenRLHF (v1.0.0), which suggests the project tracks upstream versioning rather than maintaining an independent release cadence. The news section records that as of verl v0.4.1 the feature can be enabled with a single flag, +ttrl.enable=True. If you are already running verl, that flag is the whole integration. If you are not, you are adopting verl, with everything that implies about configuration complexity and GPU scheduling.
Getting a run started
The documented path is short. Clone the repository, cd into the verl subdirectory, create a Python 3.10 conda environment, run bash scripts/install_ttrl_deps.sh, then pip install -e . from that directory. The AIME 2024 reproduction is a single command, bash examples/ttrl/Qwen2.5/aime.sh. Data has to be in Parquet; the README points to verl/data/preprocess.py for the JSON conversion. On the configuration side the flag to know is +ttrl.enable=True, which the news entry ties to verl v0.4.1 and later. The README does not enumerate the remaining keys the example scripts set, so treat aime.sh as the source of truth for the full argument list rather than expecting the README to document it. Note also the hardware line: all experiments were conducted on 8 x NVIDIA A100 80GB GPUs. Nothing in the material describes a smaller configuration.
The variance problem the README admits
One detail in the getting-started section deserves more attention than it usually gets. The authors report three independent runs using a preview version of the code: two reached pass@1 (greedy) of 43.3 and one reached 46.7. That is a spread of 3.4 points across three seeds on the same benchmark. It is honest reporting, and it is also the clearest limitation in the material. Any single TTRL run on AIME 2024 is consistent with a range of outcomes, so a one-off improvement over a baseline is not by itself evidence that the method helped. The README's headline figure, roughly 211% pass@1 improvement for Qwen-2.5-Math-7B on AIME 2024, is a much larger effect than seed noise, but the variance still defines how you would have to evaluate this in your own setting. Budget for repeats, or accept that you cannot distinguish a real gain from a lucky initialization.
Where TTRL is the wrong tool
The majority-vote reward assumes answers are discrete and comparable. Free-form generation, code that has to compile, proofs, and anything where two correct outputs look nothing alike will not produce a usable majority. A task where the model is mostly wrong at the start is also a bad fit: if the modal answer is incorrect, TTRL reinforces it. That is the failure mode the paper's framing implies and the README does not spell out, and it is the reason the reported gains are on math benchmarks rather than open-ended tasks. The second constraint is cost. Test-time training means you are generating n samples per prompt and then running gradient updates, on the evaluation set, every time you want the improvement. If your evaluation set is large or your model is large, the compute bill for inference plus training may exceed the cost of just labeling a few hundred examples and doing ordinary supervised fine-tuning. TTRL is worth it when labels are genuinely unavailable, not when they are merely inconvenient.
Compared with the OpenRLHF release in the same repo
The repository lists two releases: verl (v2.0.0) and OpenRLHF (v1.0.0). The README's current instructions are built entirely around the verl path, with the OpenRLHF tag apparently left over from an earlier implementation. The difference between the two frameworks is architectural rather than cosmetic. OpenRLHF is a standalone RLHF library with its own trainer, its own rollout workers and its own parallelism layout. verl is a hybrid-controller design where the training and generation stages are composed as separate engines, and it is the framework the TTRL authors moved to in the 2025-05-23 update, which they describe as reimplementing the code on verl. Choosing between them is really choosing which framework you already run. If you have an OpenRLHF training stack, the TTRL reward function is a small change you can port; if you have verl, the flag does it for you. The material does not indicate that the OpenRLHF path is still maintained.
Licence and the cost of tracking upstream
TTRL is MIT licensed. That is permissive, and it is also the licence of the upstream project it forks, which matters for a fork: if you vendor TTRL into a product, you are distributing code derived from verl as well as from TTRL, and you should confirm the licence of the specific verl revision you built against rather than assuming the MIT header covers everything in the tree. This is a factual point about provenance, not legal advice. On maintenance, the repository was last pushed in April 2026 and has an active news trail through March 2026, including a URLVR branch accepted to ICLR 2026. The practical cost is the opposite of abandonment: because TTRL tracks verl's release tags and depends on a specific flag introduced in v0.4.1, upgrading verl means re-checking that the flag and the example scripts still behave. Pin your verl version in the environment rather than tracking main, and re-run examples/ttrl/Qwen2.5/aime.sh after any bump to confirm the reward path still fires.
Editorial conclusion
Adopt TTRL if you have a reasoning model, a benchmark whose answers are checkable by string match, and enough GPU memory to run inference and training at the same time; the repository states that all experiments used 8 x NVIDIA A100 80GB, and the AIME 2024 reproduction is a single script. Do not adopt it if your task has no comparable answer format, since the majority-vote reward depends on answers that can be tallied. Before committing, verify three things: that +ttrl.enable=True is present in the verl version you installed, that your data is already in Parquet, and that your pass@1 variance across seeds is small enough that a single run tells you something.
Community notes