SDPO: Turning Failed Rollouts and Runtime Errors Into a Dense Training Signal

The credit-assignment gap SDPO is built to close

Standard reinforcement learning with verifiable rewards gives a model one scalar per attempt. The attempt either passed or it did not. Everything the environment knew about why it failed, the traceback, the failing test name, the judge's sentence about an off-by-one, collapses into that single number. The README frames this as a severe credit-assignment bottleneck, and that framing is the whole reason the project exists. Many verifiable environments already produce rich textual feedback, and SDPO's claim is that this text is a training signal sitting unused. The intended user is a researcher or small team already post-training a language model on code or math with an on-policy method, who has access to multi-GPU nodes and wants denser supervision without standing up a separate reward model or a larger external teacher. The README names the setting Reinforcement Learning with Rich Feedback, or RLRF, and positions SDPO as the framework for it. The second audience is narrower: people working on inference-time search, because the project also describes a test-time variant that needs no additional training.

The self-teacher: how feedback becomes a per-token gradient

The mechanism is stated plainly in the README: SDPO treats the current model conditioned on feedback as a self-teacher, and distills that model's feedback-informed next-token predictions back into the policy. There is no external teacher and no explicit reward model in the loop. The same weights play both roles, but under different conditioning, and the gap between the conditioned and unconditioned distributions is the learning signal. The README describes three granularities of credit assignment: logit, token, and sequence level. That ordering matters, because the coarser the signal, the closer you are to the scalar-reward regime SDPO is trying to escape. When environment feedback is sparse or rule-based, the README says SDPO reuses high-reward rollouts as implicit feedback instead, so a correct attempt from the same policy becomes the demonstration the teacher conditions on. The reported result is that the self-teacher improves during training and the final student surpasses the initial teacher, which is the property that makes bootstrapping from your own outputs defensible rather than circular. One design consequence worth flagging: the method depends on the model's ability to retrospectively identify its own mistakes in context. If a model cannot read a traceback and locate the faulty line, the conditioned distribution will not differ usefully from the unconditioned one, and the dense signal degrades toward noise.

Two install paths, and the hardware they assume

The README lists Linux as the operating system, tested on SLES 15 SP5 and Ubuntu 22.04, with Python 3.12 tested at 3.12.3, NVIDIA GPUs, and a CUDA driver compatible with whichever PyTorch you install. For GH200 aarch64 clusters on CUDA 13.1 there is a Dockerfile.gh200 based on the NGC vLLM container, built with podman build . -f Dockerfile.gh200 -t sdpo-gh200 and then converted for cluster schedulers with enroot import -x mount -o sdpo-gh200.sqsh podman://localhost/sdpo-gh200:latest. The note attached to that image says it uses requirements-gh200.txt, which pins versions from requirements-full.txt while excluding packages already present in the NGC vLLM container: torch, vllm, flash-attn, xformers, triton. That exclusion list is the practical detail. If you build from the GH200 requirements on a machine that is not running the NGC vLLM image, those five packages will be missing. The local path splits by GPU generation. Ampere and Hopper, meaning RTX 30 and 40 series and H100, install torch==2.5.1 with the cu124 index URL. Blackwell, meaning RTX 50 and RTX PRO 2000 Blackwell, install torch==2.7.0 with cu128. The README then says to install SDPO and core dependencies, and the supplied text is truncated mid-sentence at that point, so the exact pip invocation is not recoverable from this material. Anyone following along should read the repository's own getting-started section for that command rather than guessing.

What the reported comparisons do and do not establish

The README reports SDPO against GRPO on reasoning benchmarks, measured as highest achieved avg@16 within 1 hour and within 5 hours of wall-clock training. It states that both SDPO and on-policy GRPO take one gradient step per generation batch while GRPO performs 4 off-policy mini-batch steps, that hyperparameters were selected for 5h accuracy, and that each run used a node with 4 NVIDIA GH200 GPUs, taking approximately 6 hours including initialization and validation. Three seeds are reported with standard errors as shaded areas. Read that carefully. The headline is a wall-clock comparison at fixed hardware, not a sample-efficiency comparison, and the two methods are not doing the same amount of gradient work per batch. A reader who cares about steps-to-target rather than hours-to-target will not find that number in this material. The chemistry result is a training progression for Olmo3-7B-Instruct with average accuracy across 16 samples per question and a rolling response-length average over 5 steps. The rich-feedback result on LiveCodeBench-style data is described qualitatively, with left and right panels showing denser credit assignment and a self-teacher that improves through training. No absolute scores appear in the supplied text. The test-time result claims SDPO solves hard coding problems that neither the base model nor multi-turn interaction solves, with higher solution discovery rates across generation budgets. These are the authors' own reported numbers on their own hardware, and nothing here has been independently reproduced.

Where SDPO is the wrong tool

The requirements are the first limitation. Linux, NVIDIA GPUs, and a CUDA driver matched to a specific PyTorch build rule out CPU-only experimentation and most laptops. The reported runs used 4 GH200 GPUs per node, which is a cluster allocation, not a workstation. A second limitation is structural: SDPO needs feedback that can be tokenized and fed back into the prompt. If your environment returns only a scalar, the README's fallback is to treat high-reward rollouts as implicit feedback, which means you are now depending on the model producing at least some correct attempts to learn from. In a regime where the base model almost never succeeds, that fallback has nothing to condition on, and you are back to sparse rewards with extra machinery. Third, the self-teacher is the same model, so the method inherits its blind spots. A model that misreads a particular class of error will produce a conditioned distribution that encodes the same misreading. Fourth, there are no retrieved releases, so there is no versioned artifact to pin against and no changelog describing what changed between states of the code. Fifth, the README itself is truncated in the supplied material right where the local install command should be, which is a fair proxy for how much hand-holding to expect from the documentation.

GRPO, and the difference that actually matters

The README's own baseline is GRPO, and the comparison is instructive because the two methods differ in more than a hyperparameter. GRPO learns from group-relative scalar rewards: generate a batch, rank the attempts against each other, push probability toward the better ones. Nothing in that loop reads the text of a failure. SDPO keeps the on-policy generation but adds a second forward pass over the same model conditioned on feedback, and distills the difference in next-token predictions. GRPO's signal is a scalar broadcast across the sequence; SDPO's is a distribution over tokens. The README's own description of the rich-feedback result is that SDPO benefits from denser credit assignment at the logit, token, and sequence levels. The practical consequence is that GRPO wastes the traceback and SDPO consumes it. The cost is that SDPO needs the feedback to exist and to be well-formed, and it needs a second conditioning pass, which is why the reported comparison is framed in wall-clock hours on a fixed node rather than in gradient steps. If your environment already returns structured error text and you are running GRPO, the delta is the thing worth measuring on your own task.

Test-time self-distillation as a separate mode

The README describes a second use of the same idea that requires no training at all. The model generates multiple candidate solutions, high-quality responses are identified, and those responses are reused as demonstrations so the model refines its outputs at inference. The reported setting is hard coding problems, with the claim that SDPO solves questions neither the base model nor multi-turn interaction solves, measured by solution discovery rate across generation budgets. This is a sampling-and-selection loop with self-conditioning, and its economics are different from the training path: you trade inference tokens for accuracy, and the ceiling is set by whether any candidate in the budget is good enough to serve as a demonstration. The README does not state the selection rule in the supplied text, so how candidates are judged high-quality is something to check in the code before relying on this mode. It is also the part of the project most likely to be useful to someone who cannot get a GH200 allocation, since it needs inference only.

Maintenance, licence, and what to verify before you commit

The repository is not archived, the default branch is main, and the last push recorded is 2026-07-01, so the project is under active work rather than frozen. There are no retrieved releases, which means upgrades are tracked by commit rather than by version number, and a dependency pin in requirements-full.txt can move under you between checkouts. The licence is Apache-2.0, which permits commercial and academic use and modification and includes an explicit patent grant; the usual obligations around retaining notices and stating changes apply, and if you are redistributing a modified training stack you should read the licence text rather than this summary. The README cites arXiv paper 2601.20802 and links W&B logs at wandb.ai/jonhue/SDPO, so the training curves behind the reported figures are published rather than only described. The maintenance cost to plan for is the dependency surface: PyTorch pinned per GPU generation, a separate requirements file for the GH200 image that deliberately omits five packages, and a Docker path tied to CUDA 13.1 and aarch64. If your cluster is x86 with an older driver, the local install path is the one you are on, and the exact dependency command is the first thing to read in the repository itself.

Editorial conclusion

Adopt SDPO if you already run on-policy RLVR with GRPO-style baselines on NVIDIA hardware, have verifiable tasks that emit textual feedback such as runtime errors or judge evaluations, and can afford a node of 4 GH200-class GPUs for a run the README describes as roughly 6 hours including initialization and validation. Do not adopt it if you need a CPU-only or single-consumer-GPU loop, if your task rewards are purely scalar with no feedback text and you are not willing to treat high-reward rollouts as the implicit feedback signal, or if you need a stable released API rather than a research repository whose README is truncated mid-install-command. Before committing, verify three things in your own checkout: that requirements-full.txt resolves against the PyTorch build you installed for your GPU generation, that the Dockerfile.gh200 path matches your cluster's CUDA and architecture, and that your feedback text is actually tokenized into the prompt the way the training loop expects.

Official sources

1. Issues
2. lasgroup/SDPO on GitHub
3. License: Apache-2.0
4. Project website
5. README