NanoGPT-Speedrun-Winner: training GPT-2 to 3.28 val loss in 74 seconds on 8xH100

What the nanoGPT speedrun is and who this repository is for

The nanoGPT speedrun is a race, not a library. The target is fixed: train a GPT-2 124M model to 3.28 validation loss on a single node of 8xH100 GPUs, as fast as possible. Keller Jordan's modded-nanogpt repository defines the rules and hosts the leaderboard; this repository, NanoGPT-Speedrun-Winner, is one entry that claims to beat record #86 by about 1.3 seconds.

That framing decides the audience. The README states the goal plainly: "train a GPT-2 (124M) to 3.28 validation loss on one 8xH100 node, as fast as possible." If you are not on that hardware, the number is not reproducible for you, and the repository will not help you train a language model on a single consumer GPU. This is a benchmark harness with a training script attached. The people who get value from it are the ones already competing on the same leaderboard, or engineers studying how a small set of kernel and schedule changes moves a fixed-accuracy wall-clock number. Everyone else is reading it as a case study.

Tail EMA, lean ReLU squared and the dropped MLP residual

Three changes separate this entry from record #86, and the README ranks them by contribution. The first is a tail EMA of the readout weights: instead of using the final step's weights, the run averages the model over the last stretch of training. The README says this is "more accurate, which let us cut training from 1300 to 1288 steps," and calls it "most of the speedup." That is a schedule change, not a kernel change, and it is the kind of trick that only pays off when accuracy at a fixed loss threshold is the stopping condition.

The other two are memory-traffic reductions in the MLP. Lean ReLU squared stores only the squared activation and rebuilds the input during the backward pass rather than keeping both tensors alive. Dropping the saved MLP residual recovers that gradient "from a small identity instead of re-reading a big tensor." Both trade recomputation for bandwidth. On an 8xH100 node with the training step already short, bandwidth is the plausible bottleneck, which is why these show up in a speedrun entry at all.

The README is explicit that the rules were respected: "Same data, same validation, same rules, only the schedule, optimizer, and kernels changed." The repository layout matches that claim: train_gpt.py is the entry point, with triton_kernels.py, dc_triton_kernels.py and src/ alongside it, and a logs/ directory holding the raw run outputs.

The paired referee is the part worth copying

Speedrun claims live or die on measurement, and this repository's measurement design is more interesting than its speedup. The README describes a paired referee: the candidate and record #86 are timed back-to-back on the same 8xH100 node, so node-to-node speed variance cancels and the reported signal is delta_s = baseline_s minus candidate_s, positive meaning faster.

The README reports three paired runs in logs/paired_runs.jsonl with delta_s of +1.40, +1.26 and +1.27 seconds, and validation losses of 3.2776, 3.2775 and 3.2774. Mean delta is +1.31 seconds. For accuracy, it pools six non-cherry-picked runs across logs/paired_runs.jsonl and logs/loss_runs.jsonl, giving a mean val_loss of 3.27747 with standard deviation 0.00069, and applies modded-nanogpt's stated significance rule. The README writes that check as (3.28 minus 3.27747) times the square root of 6, which comes to 0.00621, at or above the required 0.004.

That is the strongest thing in the repository. The claim is scoped to a specific baseline on a specific node, and the raw files are committed rather than summarized. The weakness is that three paired runs is a small sample, and the README itself does not report a confidence interval on the delta, only on the loss. If you are deciding whether to trust the 1.3-second figure, the place to look is the jsonl files, not the table.

Reproducing the run: environment and the one command

There is no install section. The README gives a requirements list and a single command, and expects you to have the environment already. The stated requirements are one node with 8xH100 GPUs, PyTorch 2.10 with the CUDA 12.8 build, flash-attn 3, Triton 3.6, and FineWeb tokens from the kjj0/fineweb10B-gpt2 dataset as used by modded-nanogpt. The README says this "matches record #86's environment," so the comparison is meant to be like for like.

With that in place, the run is:

torchrun --standalone --nproc_per_node=8 train_gpt.py

The script prints val_loss and its own train_time at the final step. The README notes that the reported number is training-only time, with compile and warmup excluded, so the figure you compare against the leaderboard is the one the script prints, not your shell's wall clock. If your node is not 8xH100, or your flash-attn build is not 3, expect the run to fail or to produce a number that is not comparable, and the README offers no fallback path for either case.

Where this repository stops being useful

The limitations are structural rather than incidental. First, the hardware requirement is absolute: 8xH100 on one node. That is not a recommendation, it is the definition of the benchmark, so there is no reduced configuration to fall back on and no CPU or single-GPU path documented.

Second, the result is defined relative to record #86. The 74.19 to 74.6 second range and the 1.3-second delta only mean something against that baseline on that node. The README's own framing makes this clear: the honest signal is the paired delta, not the absolute time. A reader who quotes 74 seconds without the baseline is quoting the wrong number.

Third, the repository is not a package. There is no setup.py, no pyproject.toml in the top-level entries, and no pip install step in the README. You clone it and run train_gpt.py. There is also no licence file among the top-level entries, which matters if you intend to reuse triton_kernels.py or dc_triton_kernels.py in your own work rather than just read them.

Finally, the README does not document rollback, version pinning beyond the stated library versions, or how to run the paired referee yourself against a different baseline. The referee design is described; the tooling to reproduce it is not laid out step by step.

How it differs from modded-nanogpt itself

The obvious alternative is the upstream project this is built on: Keller Jordan's modded-nanogpt. The difference is one of role, not of technique. modded-nanogpt is the track. It defines the rules, the validation procedure, the significance threshold, and the leaderboard against which entries are ranked. NanoGPT-Speedrun-Winner is a single car on that track, and the README states the dependency directly: "Built on Keller Jordan's modded-nanogpt."

If your goal is to enter the speedrun, you start from modded-nanogpt, because that is where the rules and the current record live, and you read this repository for three specific ideas: tail EMA of the readout weights, lean ReLU squared, and the dropped MLP residual. If your goal is to understand how a leaderboard entry is validated, modded-nanogpt gives you the referee protocol and this repository gives you a worked example of applying it, including the arithmetic for the significance test. Neither is a general-purpose training framework, and choosing between them is not a matter of quality; it is a matter of whether you want the rulebook or one competitor's notebook.

Maintenance, licensing and what to check before relying on it

The repository is not archived, and the last push was on 2026-09-04. That is recent enough that the code has not visibly gone stale, but the README describes a fixed result against a fixed baseline, and speedrun entries age by being beaten rather than by being abandoned. If record #86 is superseded, the comparison in this README becomes a historical one, and the delta numbers stay valid only as a record of that pairing.

On licensing, the repository carries no licence identifier. The README credits modded-nanogpt as the base, so if you plan to copy code out of triton_kernels.py or dc_triton_kernels.py into another project, check the licence of both repositories yourself. Nothing here should be read as legal advice, and the absence of a licence file is a fact to resolve before reuse, not a permission.

On upgrade cost, the pinned environment is the constraint. PyTorch 2.10 with the CUDA 12.8 build, flash-attn 3 and Triton 3.6 are stated as matching record #86. Moving any of those versions forward means your run is no longer the same measurement, and the README does not describe how the numbers behave under different versions.

Editorial conclusion

Adopt this if you already run modded-nanogpt on an 8xH100 node and want a reference implementation of tail EMA plus the lean ReLU squared and dropped-residual memory tricks; the paired referee logs in logs/paired_runs.jsonl are the part worth reading even if you never run train_gpt.py. Do not adopt it as a general GPT-2 training script, as a way to train on smaller hardware, or as a packaged library: the README gives one torchrun command and no install instructions, no licence file is present in the repository, and the whole result is defined against a single 8xH100 node. Before trusting anything, open logs/paired_runs.jsonl and logs/loss_runs.jsonl and check that each row carries both candidate_s and baseline_s from the same node, because a delta computed across two different nodes is not the measurement this project claims to make.

Frequently asked questions

Official sources

1. colinlikescode/NanoGPT-Speedrun-Winner on GitHub
2. Issues
3. README