# NanoGPT Slowrun: a language modelling benchmark where the clock stops mattering

> Slowrun inverts the usual speedrun format. The data is fixed at 100M tokens from FineWeb, there is no time limit on the open track, and the only measure is validation loss, which turns a training script into a public leaderboard of algorithm ideas.

**qlabs-eng/slowrun** — 100M tokens. Infinite compute. Lowest val loss wins.

- Repository: https://github.com/qlabs-eng/slowrun
- Stars: 571 · Forks: 88
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/qlabs-eng-slowrun

## Fixed data, no clock, lowest loss wins

NanoGPT Slowrun is described as a new benchmark for language modelling algorithms in the infinite compute, fixed data regime: 100M tokens from FineWeb, no compute or time limit on the open track, and the lowest validation loss wins. The name is the argument. A slowrun is the opposite of a speedrun, and the README says the goal is to spend as much time with the data as is needed to maximise learning on it. It is deliberately contrasted with speedruns like modded-nanogpt, which assume infinite data and optimise for wall-clock time on fixed hardware. The framing sentence in the README is that when speed is not the binding constraint, the space of promising algorithms changes dramatically, with large models trained with heavy regularisation, expensive optimisers and evolutionary search all becoming fair game, and the stated ambition is leaps like GPT-3 where previously unimaginable compute led to better generalisation. That last point is the one that matters, because a time budget quietly turns an architecture question into an engineering question.

## Four tracks, and the directories that hold them

There are four tracks and the compute budget is what distinguishes them. The limited compute track is capped at a single 8xH100 node for one hour, which the README notes is 100 times the compute used by the Nanochat one-epoch baseline. The tiny compute track is capped at a single 8xH100 node for 15 minutes. The two hour track is again a single 8xH100 node for two hours. And the unlimited compute track has minimal restrictions on hardware or time. Each track lives in its own directory, with the limited track in the root, the tiny track under tiny/, the two hour track under two_hour/ and the unlimited track under unlimited/. The README also says the approaches pursued vary significantly based on the compute budget, which is the point of separating them. A technique that needs 40 hours of compute is a different result from one that needs 40 minutes, and the leaderboard keeps them apart.

## The baseline, and what the record history actually shows

The baseline trains in about 47 minutes on 8xH100, at a cost the README puts at roughly 12 dollars, and reaches 3.402 validation loss. It is a 2.7B transformer with the Muon optimiser, dropout 0.1 and weight decay 1.6. The record history is a list of changes and their effect, and it reads like a catalogue of the ideas people are trying. Early entries shuffle every epoch, change the value embedding tables to projections, adopt the SwigLU activation, add a U-Net architecture, and add gating per attention head. Later ones repeat layers 15 to 20 for the last three epochs, add exclusive self attention, tune the learning rate and the warmdown, add an exponential moving average of weights, use a weighted average of the last three epoch checkpoints, add stochastic weight averaging, switch the c_proj initialisation from zero to normal, add stochastic depth training, add multi-token prediction loss, add interleaved head attention, add document-level shuffling, and add a weight decay schedule. The losses fall from 3.402 to below 3.2 over roughly two months of entries, and the run times creep up from 47 minutes to close to 60, which is the one-hour cap doing its job.

## Reproducing the current record, and what it needs from you

The limited compute record can be reproduced with a short sequence of commands, and the README asks you to set two environment variables first.

```bash
# Set HF_TOKEN and WANDB_API_KEY in your environment first
git clone https://github.com/qlabs-eng/slowrun.git && cd slowrun
pip install -r requirements.txt
python prepare_data.py
torchrun --standalone --nproc_per_node=8 train.py
```

The Hugging Face token is for the data and the Weights and Biases key is for the logging, and the eight-process launch matches the 8xH100 node the track is defined against. The requirements file is short: torch pinned to 2.10, wandb, tiktoken, numpy, datasets, tqdm and kernels. The data preparation step is separate from training, which is worth noting because it means the tokenised 100M token dataset is produced once and then reused, so two submissions are comparing training and not data preparation.

## How a submission is judged

The acceptance rule is simple and the consequence is not. The README says pull requests that achieve a new world record validation loss within the track's time limit are accepted, and an entry is added below for each improvement. Every row in the history table carries the loss, a short description of the change, a date, the run time, a link to the training script at a specific commit, and the contributors. Linking the script at a commit rather than a branch is the detail that makes the leaderboard verifiable, since the whole point of a public record is that a reader can rerun it. The README is also explicit that this is community work, with entries contributed by a long list of individuals between February and April 2026, and it notes the benchmark was liked by Karpathy. The repository root carries a val loss animation, a prepare_data.py script, and directories for data efficiency, research and the three non-default tracks, which suggests the same infrastructure is being reused for follow-on questions beyond the leaderboard itself.

## Conclusion

Slowrun fits a researcher who wants to test whether an algorithmic idea actually helps when compute is not the limit, and who is willing to spend a GPU night per attempt. It does not fit a team looking for a fast training recipe or a reproducible production training setup, because the ranking is by loss and the winning scripts differ in kind from run to run. Verify first which track you are entering, since the compute cap changes what counts as a fair comparison, and read the record history rather than the baseline alone, because the interesting information is in how the loss was driven down rather than where it started.

## FAQ

### What is the NanoGPT Slowrun benchmark?

It is a benchmark for language modelling algorithms in the infinite compute, fixed data regime: 100M tokens from FineWeb, no compute or time limit on the open track, and the lowest validation loss wins. The baseline trains in about 47 minutes on 8xH100 and reaches 3.402 validation loss.

### What are the tracks in Slowrun?

Four. The limited compute track is capped at a single 8xH100 node for one hour, the tiny track at a single node for 15 minutes, the two hour track at a single node for two hours, and the unlimited track has minimal restrictions on hardware or time. Each lives in its own directory in the repository.

### How do I reproduce the current record?

Set HF_TOKEN and WANDB_API_KEY in your environment, then clone the repository, install the requirements, run prepare_data.py to build the tokenised dataset, and launch training with torchrun across eight processes with the train.py script.

### How are Slowrun submissions accepted?

Pull requests that achieve a new world record validation loss within the track's time limit are accepted, and each improvement is added to the record history with the loss, a description of the change, the date, the run time, a link to the training script and the contributors.

## Sources

- [Issues](https://github.com/qlabs-eng/slowrun/issues)
- [License: MIT](https://github.com/qlabs-eng/slowrun/blob/main/LICENSE)
- [qlabs-eng/slowrun on GitHub](https://github.com/qlabs-eng/slowrun)
- [README](https://github.com/qlabs-eng/slowrun/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/qlabs-eng-slowrun