Model or dataset
KellerJordan/modded-nanogpt avatar
KellerJordan/modded-nanogpt

Modded-NanoGPT: the NanoGPT speedrun that trains a 124M model in under 40 seconds

NanoGPT (124M) in 90 seconds

6,041 stars928 forksPythonMIT

At a glance

What is it?
Modded-NanoGPT is a competitive benchmark repo that searches for the fastest way to train a 124M-parameter language model to 3.28 FineWeb validation loss on 8 H100 GPUs. The current record is under 40 seconds, and the README lists the techniques that got it there.
Who is it for?
Adopt Modded-NanoGPT if you have 8 H100 GPUs and want to study or contribute to the fastest known training algorithm for a 124M model. Do not adopt it if you lack that hardware, need a production training pipeline, or want a general-purpose framework.
Can I use it commercially?
Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 48 days ago.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • KellerJordan/modded-nanogpt6,041 ★ · +222
Star history
Repository2026-09-232026-11-15
KellerJordan/modded-nanogpt5.8k6k

222 Stars in the last 53 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

What the NanoGPT speedrun actually measures

The repository hosts the NanoGPT speedrun, a collaborative and competitive search for the fastest algorithm that trains a language model to 3.28 cross-entropy loss on the FineWeb validation set using 8 NVIDIA H100 GPUs. The target comes from Andrej Karpathy's GPT-2 replication in llm.c, which reached that loss after 45 minutes. The speedrun code descends from llm.c's PyTorch trainer, which itself descends from NanoGPT, which explains the repository name.

The audience is narrow and specific. This is for researchers and engineers who want to test training optimizations under a fixed, reproducible target. It is not a library you import into a product. The README states that the current algorithm attains the target in under 40 seconds on 8xH100 and under 330M tokens, compared with 45 minutes and 10B tokens for the llm.c GPT-2 replication. There is also an optimization track in records/track_3_optimization that minimizes steps subject to fixed architecture, data and batch size with an unlimited wallclock budget. The main track and the optimization track measure different things.

The techniques stacked into train_gpt.py

The improvement comes from a long list of changes the README enumerates. On the architecture side: rotary embeddings, QK-Norm, ReLU², zero-initialized projections in a muP-like scheme, skip connections from the embedding to every block and from block 3 to 6, extra embeddings mixed into attention values, a Smear module for one-token lookback, a hashed bigram and trigram embedding table with a sign trick, sharded across GPUs, MUDD skip connections, learnable XSA, paired head attention, and multi-token prediction. Training-side changes include the Muon optimizer, FP8 for the head, the MLP forward and backward passes and the QKV projections, asymmetric rescale and softcap logits, gradient accumulation for two steps on the embedding and lm_head, a batch size schedule, a max sequence length schedule, cautious weight decay tied to the learning rate, untied embedding and lm_head at two thirds of training, and a prefix token prediction auxiliary loss. Flash Attention 3 with a long-short sliding window pattern, window size warmup with YaRN, and aligning batch starts with EoS round out the list.

That is a lot of interacting changes, and the repository does not present them as independent. The README lists them as the accumulated result of many contributors, and the world record history section tracks the progression. If you want to isolate the effect of one technique, this repository is not organized for that. It is organized to find the fastest combination.

Installing Modded-NanoGPT and running the current record

The README's path to run the current record also installs the pinned torch==2.10 build from the cu128 index after the requirements, a step not shown in the block below. The clone and install steps are standard. The data script downloads only the first 900M training tokens to save time, and run.sh launches training.

bash
git clone https://github.com/KellerJordan/modded-nanogpt.git && cd modded-nanogpt
pip install -r requirements.txt
python data/cached_fineweb10B.py 9
./run.sh

If ./run.sh fails with torchrun: command not found, the README says to add torchrun to your path. The README also warns that torch.compile will add around 7 minutes of latency the first time you run the code, so a cold start is not representative of the steady-state speed.

For systems where CUDA or NCCL versions are not compatible, the README recommends Docker as an alternative that standardizes CUDA, NCCL, CUDNN and Python versions. An NVIDIA driver must already be installed on the host. The Dockerfile builds from nvidia/cuda:13.1.1-cudnn-devel-ubuntu24.04 with Python 3.12.7 and installs the requirements plus the pinned torch==2.10 build from the cu128 index.

bash
git clone https://github.com/KellerJordan/modded-nanogpt.git && cd modded-nanogpt
sudo docker build -t modded-nanogpt .
sudo docker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt python data/cached_fineweb10B.py 8
sudo docker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt sh run.sh

The Docker path downloads 8 units of data rather than 9, which is a detail worth noticing if you compare timings between the two routes. For an interactive container, the README gives a bash variant of the same run command. Official records are timed on 8 NVIDIA H100 GPUs from PrimeIntellect, which has sponsored recent validation runs.

Where Modded-NanoGPT breaks down

The first constraint is hardware. The target and the record are defined on 8 NVIDIA H100 GPUs. Running on fewer GPUs, on A100s, or on consumer cards changes the comparison entirely. If you do not have access to that hardware, you cannot reproduce the record.

The second constraint is that this is a benchmark, not a training framework. The training script is a single file, train_gpt.py, with a medium variant alongside it. If your goal is to train a model you intend to deploy, you are looking at the wrong artifact. This repository exists to answer one question: how fast can this specific target be reached?

The third constraint is timing methodology. The README notes that torch.compile adds around 7 minutes of first-run latency, and that official records are timed on PrimeIntellect hardware. If you time a run on your own cluster without accounting for compilation and without matching CUDA and NCCL versions, your number will not be comparable to the leaderboard. The Docker path exists precisely to reduce that variance, but it does not eliminate differences in the underlying hardware.

How this differs from NanoGPT and llm.c

NanoGPT, the project this one descends from, is a clean, readable implementation meant for learning and for reproducing GPT-2 training. It prioritizes clarity. Modded-NanoGPT inverts that priority: the training script is the accumulated output of dozens of contributors stacking micro-optimizations, and readability is not the goal. If you want to understand how transformer training works, read NanoGPT. If you want to see how far the training loop can be pushed, read this repository.

llm.c is the other reference point. Andrej Karpathy's GPT-2 replication in llm.c reached 3.28 validation loss after 45 minutes and 10B tokens. Modded-NanoGPT reaches the same loss in under 40 seconds and under 330M tokens. The difference is not one trick. It is the full list of architecture, optimizer, precision and systems changes applied together. A reader who wants a single change to adopt should look at the Muon optimizer, which has its own writeup and repository linked from the README, rather than trying to port the whole stack.

Maintenance, licence and upgrade cost

The repository is not archived, and the last push was on 2026-09-28. That is recent enough that the code is moving, and the README's contributor list grows with each new record. The practical consequence is that the record changes. A technique that is in train_gpt.py today may be superseded, and the world record history section exists to track that progression. If you fork the repository to run your own experiments, expect to rebase against a moving target.

The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are included. That is a permissive licence, but it applies to the code, not to the FineWeb dataset or to any model weights you train. Check the FineWeb dataset page before using it in a commercial context. This is not legal advice.

The upgrade cost is mostly dependency churn. requirements.txt pins torch==2.10 and typing-extensions==4.15.0, and the Dockerfile installs the same pinned torch==2.10 from the cu128 index, warning that a nightly has produced NaNs in reproduction attempts. If you need reproducibility, the Dockerfile is the more controlled route because it fixes the CUDA 13.1.1 base, Python 3.12.7 and torch==2.10 together, so treat the container as the reference environment.

Editorial conclusion

Adopt Modded-NanoGPT if you have 8 H100 GPUs and want to study or contribute to the fastest known training algorithm for a 124M model. Do not adopt it if you lack that hardware, need a production training pipeline, or want a general-purpose framework. Before running, verify that your CUDA and NCCL versions are compatible, since the README recommends Docker precisely for that case, and check whether torch.compile's roughly 7-minute first-run latency fits your timing budget.

Frequently asked questions

What hardware does Modded-NanoGPT require?

The speedrun target is defined on 8 NVIDIA H100 GPUs, and official records are timed on that hardware from PrimeIntellect.

How do I install and run Modded-NanoGPT?

Clone the repository, run pip install -r requirements.txt, install torch==2.10 from the cu128 index, download data with python data/cached_fineweb10B.py 9, then execute ./run.sh. The README notes that torch.compile adds around 7 minutes of latency on the first run.

What is the difference between the main track and the optimization track in Modded-NanoGPT?

The main track minimizes wallclock time to reach 3.28 FineWeb validation loss on 8 H100s. The optimization track in records/track_3_optimization minimizes steps subject to fixed architecture, data and batch size with an unlimited wallclock budget.

Official sources

  1. Issues
  2. KellerJordan/modded-nanogpt on GitHub
  3. License: MIT
  4. README
Community

Where developers are discussing it

Posts on Hacker News, dev.to and Lobsters that link to this repository, found by our scan of those communities. 0 in the last 30 days; newest first.

Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/kellerjordan-modded-nanogpt.svg)](https://hysenlabs.com/projects/kellerjordan-modded-nanogpt)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/kellerjordan-modded-nanogpt.svg)](https://hysenlabs.com/projects/kellerjordan-modded-nanogpt)

Was this useful?