DFT: scaling token loss by probability to fix SFT generalization

The change is one line of loss code, the argument is about reward structure

Dynamic Fine-Tuning, abbreviated DFT, is presented as a theoretically motivated improvement to supervised fine-tuning for large language models, aimed at the limited generalization that SFT shows compared with reinforcement learning. The argument runs through the gradients themselves. According to the abstract, standard SFT gradients implicitly encode a problematic reward structure, and that structure is what restricts how well the model generalizes rather than how well it fits.

The remedy is to rectify that reward structure by rescaling the objective per token with the probability of that token, which stabilizes the gradient update. The implementation is described as a single line of code and a single-line change to a standard SFT setup.

loss = loss * torch.softmax(shift_logits, dim=-1).gather(1, shift_labels.unsqueeze(-1)).squeeze(-1).detach()

Read that line carefully and the whole method is visible. The probability the model assigned to the target token is taken from the shifted logits, gathered at the label position and detached so that no gradient flows through the weight. Multiplying the token loss by that probability means that tokens the model already finds easy, which carry a high probability, contribute proportionally less to the update, and tokens the model finds surprising contribute more. The detach is what keeps this from turning into a second gradient path through the softmax.

The repository is a patch on top of the verl training stack

The repository is thin at the top level: README.md, a math_evaluation directory and a verl directory. That is the whole inventory, and it tells you what kind of project this is. DFT is not a standalone fine-tuning script with its own trainer loop. It is a set of training and evaluation scripts laid over the verl reinforcement learning training stack, and the launcher used in the documentation is a module inside that stack.

-m verl.trainer.fsdp_dft_trainer

That naming tells you several things at once. The trainer is an FSDP based trainer, so training is sharded across GPUs, and the module name carries dft, so a DFT run is launched through a dedicated entry point rather than through a flag on a generic trainer. The math_evaluation directory is separate again, which matches the paper's structure, where evaluation is treated as its own pipeline over math benchmarks.

The practical consequence for an adopter is that you inherit the stack. You get FSDP sharding, the data format conventions of the stack, its configuration system, its logging and its checkpoint layout, and you have to fit your own data into those conventions rather than inventing your own.

Setting up the environment means installing vllm, sglang and mcore

The documented environment was tested on H100 servers with python 3.10.0 and torch 2.6.0+cu124. The install path is a conda environment, a helper script that pulls in the inference and training dependencies, and then an editable install of the project without dependencies, which keeps pip from resolving versions over the pinned set.

git clone https://github.com/yongliang-wu/DFT.git
cd DFT

conda create -n DFT python=3.10 -y
conda activate DFT
cd verl
bash scripts/install_vllm_sglang_mcore.sh
pip install --no-deps -e .

Two observations. The helper script name is the shortest description of the dependency set: vllm, sglang and mcore, which are the serving and model parallel components needed for a training run of this size, and you are expected to run it rather than to pick versions yourself. And the editable install with --no-deps means the environment is exactly what that script produced plus what you add, so if you hit a version conflict the helper script is the place to look first.

The clone step is separate from the environment step, and the editable install happens from inside the verl directory, which is where the package lives.

Data preparation produces a parquet training set and a math500 split

Two scripts prepare the data, and they do different jobs. The training script generates the training data, and its help text notes that the volume is controlled by a --train_end argument, which the example sets to 100000. The evaluation script generates the evaluation data, with no extra flags in the example.

# Generate training data (optional: change --train_end to control volume)
python examples/data_preprocess/numina_cot.py --train_end 100000

# Generate evaluation data
python examples/data_preprocess/math_dataset.py

The training script name points at the source, a chain of thought dataset processed from Numina, and the evaluation script produces a math500 test set that the training command then points its validation files at.

Both scripts live under examples/data_preprocess inside the verl tree, so they follow the same pattern as the training entry point. The format of the resulting data is visible in the training configuration rather than in the scripts, where the prompt and response are read from the same key, extra_info, and the fields inside it are selected by data.prompt_dict_keys and data.response_dict_keys.

Launching a run is one torchrun command with FSDP settings

The training command sets a handful of shell variables first, then hands them to torchrun as configuration overrides. The example uses eight processes on one node and names the experiment after the dataset and the base model.

nproc_per_node=8
project_name=numina-cot

experiment_name=numina-cot-dft-qwen-2.5-math-1.5b
save_path=checkpoints/$experiment_name

The trainer itself is launched as a module, with the data paths, the model and the optimisation settings passed as key value pairs.

torchrun --standalone --nnodes=1 --nproc_per_node=$nproc_per_node \
        -m verl.trainer.fsdp_dft_trainer \
    data.train_files=data/numina_cot/train.parquet \
    data.val_files=data/math500/test.parquet \
    data.prompt_key=extra_info \
    data.response_key=extra_info \
    data.train_batch_size=256 \
    data.max_length=2048 \
    optim.lr=5e-5 \

The second half of the command covers the model and the trainer, and it is where the practical choices sit.

    data.prompt_dict_keys=['question'] \
    data.response_dict_keys=['answer'] \
    data.micro_batch_size_per_gpu=4 \
    model.partial_pretrain=Qwen/Qwen2.5-Math-1.5B \
    model.use_liger=True \
    model.fsdp_config.model_dtype=bf16 \
    trainer.default_local_dir=$save_path \
    trainer.project_name=$project_name \
    trainer.experiment_name="$experiment_name-$(date +%Y%m%d-%H%M%S)" \
    trainer.logger=['console','tensorboard'] \
    trainer.total_epochs=1 \

The base model in the example is a 1.5 billion parameter math model, the micro batch per GPU is 4 and the global training batch is 256, so the run expects eight GPUs. Logging goes to the console and to tensorboard, the checkpoint path is the directory you set, and the experiment name is timestamped so that repeated runs do not overwrite each other. The use_liger flag enables a fused kernel path, and the dtype is bf16.

The evidence stops at math data and 7 billion parameter models

The authors are unusually explicit about the limits, and their own community-feedback section is the most useful part of the repository for an adopter. They report hearing about both successes and failures when people applied DFT, for instance in literary or financial tasks, and they state plainly that they do not claim DFT can replace SFT in all cases.

The limitation they quote from their own work is that the experiments demonstrate substantial gains on mathematical reasoning benchmarks, and that this evaluation is confined to math focused datasets, with code focused datasets to be released in a later version, and models up to 7 billion parameters. The example training command uses a 1.5 billion parameter model, which is consistent with that ceiling.

The timing of the release is worth noting too. The news entry records 2025.08.08 as the date when the training scripts, evaluation scripts and model checkpoints were released, and the checkpoints are hosted as a Hugging Face collection rather than in the repository, which is where a Hugging Face models badge in the README points.

No releases, no licence file, and an upstream stack you also depend on

The maintenance picture for a research code drop is different from that of a library, but it still needs reading.

The last push was on 2026-08-01 and the repository has no GitHub releases, so there is no tag to pin a training environment to and no release notes describing what changed. The version of the underlying stack you train against is whatever verl happens to be in the tree on the day you clone, which is the real upgrade cost here: the moving part is the framework the experiment runs on, not DFT itself, whose footprint is one line.

The licence field for this repository carries no value and there is no LICENSE file among the top level entries, which is normal for a paper repository but still something to resolve before you use the code or the released checkpoints inside a commercial pipeline.

The last point is about alternatives. The paper positions DFT against two things: plain SFT, which is what you run when you apply the standard token loss with no probability weighting, and reinforcement learning, where the same generalization benefit is obtained by actually optimising against a reward. The claim is that DFT reaches competitive results in offline RL settings with a simpler method and a fraction of the machinery. If your budget allows full RL training on the same data, that is the alternative to compare against, and the comparison is the experiment the repository makes easy, because the same trainer stack and the same data pipeline serve both.

Editorial conclusion

Adopt DFT when you are fine-tuning a base model on math or code data and want the cost of an SFT run, with a change you can inspect in one line, and when you can compare against a plain SFT baseline on your own distribution. Do not adopt it on the strength of the paper for literary or financial fine-tuning, since the authors state they have seen both successes and failures in those settings and that the evaluation is confined to math focused datasets and models up to 7 billion parameters. Before the first run, check that your hardware matches the tested environment of python 3.10.0 with torch 2.6.0+cu124 on H100 servers, generate the evaluation split with math_dataset.py so you have a baseline, and read the licence situation yourself, because the repository carries no licence value and no LICENSE file among its top level entries.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. yongliang-wu/DFT on GitHub