Rethinking OPD, a study of why on-policy distillation sometimes fails and how to recover it

The claim, and the two conditions behind it

On-policy distillation has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood, and that gap is what this paper sets out to close. The paper first identifies two conditions that govern whether OPD succeeds or fails. The student and teacher should share compatible thinking patterns. And even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. The second condition is the one that makes the paper more than a tuning guide, because it says a better teacher is not sufficient: a teacher that scores higher but reasons the same way as the student has nothing to teach. The findings are validated through weak-to-strong reverse distillation, where same-family 1.5B and 7B teachers turn out to be distributionally indistinguishable from the student's perspective, which is the sharpest statement of the claim in the whole project.

(i) the student and teacher should share compatible thinking patterns; and
(ii) even with consistent thinking patterns and higher scores, the teacher
must offer genuinely new capabilities beyond what the student has seen
during training.

The token-level mechanism

The paper then probes into the mechanism at the token level, and the finding is specific. Successful OPD is characterised by progressive alignment on high-probability tokens at student-visited states, and by a small shared token set that concentrates most of the probability mass, between 97 and 99 percent. That picture explains the failure mode as well as the success one. If the teacher's advantage lives mostly in a tail of low-probability tokens that the student never visits, then on-policy sampling never reaches the region where the teacher is better, and the training signal carries no information. The shared token set is the thing the diagnostics are built around, which is why the top-k overlap diagnostics the authors contributed to verl are about an overlap ratio and a token advantage. Those metrics were merged into verl, adding distillation/overlap_ratio and distillation/overlap_token_advantage, following the token-level analysis.

progressive alignment on high-probability tokens at student-visited states,
a small shared token set that concentrates most of the probability mass

Two ways to recover a failing run

From the analysis the paper proposes two practical strategies to recover failing OPD, and both are about changing what the training loop sees rather than about tuning the loss. The first is off-policy cold start, which means beginning from data that is not generated by the student, so the student visits the teacher's better regions before the on-policy phase begins. The second is teacher-aligned prompt selection, choosing the prompts so the student and teacher are reasoning about the same things rather than drifting apart. The paper also raises a question it does not fully answer: OPD's apparent free lunch of dense token-level reward comes at a cost, which raises the question of whether OPD can scale to long-horizon distillation. The project has clearly been used, with the insight on OPD adopted in MiniCPM5-1B, and the paper was accepted to the ICML 2026 FoGen Workshop.

We further propose two practical strategies to recover failing OPD:
off-policy cold start and teacher-aligned prompt selection.

The environment, and the two frameworks underneath

The code is built on two upstream projects rather than being standalone, and the environment setup reflects that. The main code is based on verl at version 0.7.0, and the setup creates a conda environment with Python 3.12, runs the verl install script with Megatron disabled, and installs math-verify. Separately, LLaMA-Factory at version 0.9.5 is used for the supervised fine-tuning stage, with its own conda environment on Python 3.11, an editable install and the metrics requirements. The repository carries both of them as directories, along with a datasets/ directory, a scripts/ directory, a verl_example/ directory with the launch script, a grpo.sh script for the reinforcement learning baseline, and the on_policy_distillation.sh script that is the main entry point. The overall shape is a research codebase that vendors its dependencies, which is normal for a paper release and means the versions are pinned by directory rather than by a lockfile.

conda create -n verl python==3.12
conda activate verl
cd verl/
USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh
pip install math-verify

The knobs, and what they change

The launch script is documented with a table of parameters, and the interesting ones are the Top-K and weighting strategy. LOG_PROB_TOP_K defaults to 16, the number of top tokens retained when computing token-level rewards, and setting it to zero falls back to sampled-token OPD. TOP_K_STRATEGY defaults to only_stu, meaning the top-K set is selected from the student and the teacher is then queried for the corresponding log-probabilities; the alternatives are only_tch, intersection, union and union-intersection, which is the symmetric difference. REWARD_WEIGHT_MODE defaults to student_p, with teacher_p and none as the other options. The generation defaults are four responses per prompt, a 1024 token prompt limit and 7168 for responses. The distillation method itself is fixed: the table says the estimator cannot be modified if you use OPD, with token_reward_direct as the default.

| `LOG_PROB_TOP_K` | `16` |
| `TOP_K_STRATEGY` | `only_stu` |
| `REWARD_WEIGHT_MODE` | `student_p` |
| `N_RESPONSES` | `4` |

The verl validation bug, and the data overlap note

One note in the documentation is worth more than most of the rest, because it will silently corrupt your results if you miss it. The built-in validation path in verl 0.7.0 can substantially under-estimate model performance, typically by 5 to 7 percentage points in the authors' runs. The issue has been fixed in verl 0.8.0. If you are reproducing the experiments with 0.7.0, the recommendation is to set the validation response length equal to the training response length, disable in-training validation with the trainer test frequency set to minus one, and run final validation separately with the evaluation scripts under scripts/val/, and the corresponding verl launch script is in the verl_example directory. A second, smaller note explains that scripts/infer/dedup_deepmath.py can be used to deduplicate DeepMath against DAPO-Math-17K to avoid data overlap, as in the experiments in section 5.2. Both notes are the kind of detail that only appears when the authors have already hit the problem themselves.

we recommend setting `MAX_VAL_RESP_LENGTH=MAX_RESP_LENGTH` and
disabling in-training validation with `trainer.test_freq=-1`

How to think about this project

The framing of the paper, phenomenology, mechanism and recipe, is worth taking seriously because it maps onto three different things a practitioner needs. Phenomenology is the observation that OPD sometimes fails and that the failure is diagnosable. Mechanism is the token-level account of why, with the shared high-probability token set as the explanatory object. Recipe is the two recovery strategies, which are usable without accepting the whole theory. That separation is what makes the repository worth reading even if you never train with it, because the diagnostic frame generalises. If your distillation run is producing a student that does not beat the teacher, the two conditions give you two questions to ask: does the teacher think differently from the student, and does it know anything the student has not already seen in its own training data. The top-k overlap metrics now in verl are the operational form of those questions, and the honesty about the verl validation bug is the reason to trust the reported numbers.

bash on_policy_distillation.sh

Editorial conclusion

Adopt the code in this repository if you are running on-policy distillation and want the paper's diagnostics and the two recovery strategies rather than the default recipe, since the work's whole point is that the default recipe fails under identifiable conditions. Skip it if you are doing supervised fine-tuning only, because the recipe here assumes a teacher model providing token-level rewards. Verify three things before you start a run: which verl version you are on, since the built-in validation path in v0.7.0 substantially underestimates performance and the fix landed in v0.8.0; whether your student and teacher share a thinking pattern, since that is one of the two conditions the paper identifies as governing success; and which Top-K strategy you choose, because the default selects from the student and the alternatives change what the teacher is even asked about.

Frequently asked questions

Official sources

1. Issues
2. README
3. Thinking-Space/Rethinking-OPD on GitHub