Parameter Golf: 16MB, ten minutes, and a leaderboard measured in bits per byte

The 16MB artifact and the 10-minute budget are both fixed

Two numbers define the task: the model has to fit in a 16MB artifact, and it has to train in under ten minutes on 8xH100s. Leaderboard submissions are held to that compute limit, which the project explains is there to keep the challenge from becoming inaccessibly expensive. Anyone who wants to ignore the limit still has a place to submit, in a section called Non-record Submissions, and the stated hope is that people push the frontier of parameter-constrained performance beyond the record line. In scaling-law terms the framing is L(N) optimization: lowest loss given a fixed parameter count, with data, compute, steps and architecture left open. The two challenges it sits beside optimize different axes, one going after lowest time given constrained loss and the other after lowest loss given a constrained dataset size. What the artifact limit rules out is anything that needs the weights to grow, which is why the leaderboard is full of compression and tying tricks rather than bigger matrices.

Bits per byte is the metric, so tokenizers compete on equal footing

Evaluation happens by compression on the FineWeb validation set, measured in bits per byte, and the measurement is described as tokenizer-agnostic. That single choice shapes everything else. Because the score is per byte of text rather than per token, an entry is not rewarded for a tokenizer that happens to emit fewer tokens for the same content. Novel tokenizers are listed among the compression schemes the challenge hopes to see, and a token-only n-gram plus asymmetric logit stack sits at the top of the leaderboard. Byte-sidecar accounting appears further down the board, where a lossless case operation transform from an earlier entry is paired with byte-sidecar bits-per-byte accounting. If you are judging a submission by its loss number, remember that the number is a compression ratio on validation text, not a capability claim about the model.

Every accepted score is a multi-seed mean with a p-value attached

The leaderboard columns are more than a name, a score and a date. Each row names the pull request it came from, and many carry a multi-seed mean, a p-value against the entry they beat, and sometimes a rerun reference. One entry reports a mean of 1.05651 with p=0.014 against the runner-up, another reports 1.05855 with p=0.063 against the base it extends, and a third reports 1.06108 with p=0.188 against the previous rerun, which is a weak signal that the gap is real. Two separate rerun references appear: a strict seed-42 rerun after a mixed-precision quantization change, and compliance reruns collected on a shared pull request. Grace policy gets named as well. Read together, these mean a single run is not a submission. For a reader, the summary column is the record; the score column alone overstates the separation.

Ten entries, 0.009 bits per byte, mostly stacked on the same base

The visible range runs from 1.0565 on 2026-05-01 to 1.0655 on 2026-04-19, a spread of 0.009 bits per byte across the nine entries with visible scores, and the author names repeat. Most of the top ten rows build on the same case-operations (CaseOps) lineage, and several extend pull request 1787, which itself added Polar Express Newton-Schulz coefficients with a minimum learning rate set to 0.1. The technique vocabulary reads like a list of failed experiments that partly worked: a smearing gate, LQER, a sparse attention gate, fused softcapped cross-entropy, gated attention with quant-gate scaling, phased test-time training, warm-started test-time training, asymmetric logits, an AWQ-lite mixed GPTQ pass, an MLP clip (MLPClip12), and per-group lrzip compression. One row admits to nine greedy hyperparameter overrides. What the board does not show is a clean attribution, since almost every entry is a sum of other entries, and the composition of any single gain is not isolated.

Compute credit requires a form, a justification and a linked account

The challenge is sponsored with 1,000,000 dollars in compute credits, and the request route is a named form linked from the project page. Three conditions are stated for a request: pick the appropriate level, write sufficient justification, and submit with an email tied to an OpenAI or ChatGPT account. The third condition is the one that shapes who can apply, because an unrelated address does not qualify. A separate participant form exists for self-introduction, hosted on an OpenAI jobs form, and it is explicitly not required to participate. That form is tied to a stated hiring plan: a small cohort of early-career researchers in June, targeting current undergraduates and recent graduates, including Olympiad medalists and elite competitors. The framing names mathematics and programming competitions as the model, which tells you what kind of submission the organizers are reading for. The challenge window itself is short, running from March 18 to April 30.

The repository ships scripts and data, not an installable artifact

There is no package to install and no release to download. The repository has no GitHub releases, so a participant clones the tree, which holds `train_gpt.py` as the main training entry point, `train_gpt_mlx.py` alongside it for a second backend, and `measure_variance_ratio.py` for working out run-to-run noise. Directories cover `data/`, `scripts/`, `results/`, `records/` and `paper/`, and the licensing side is covered by `LICENSE` plus `THIRD_PARTY_NOTICES.md`. The dependency set is pinned almost entirely open, with a single exact pin:

numpy
tqdm
torch
huggingface-hub
kernels
setuptools
typing-extensions==4.15.0
datasets
tiktoken
sentencepiece

Nine of those ten dependencies float on whatever version resolves, so two submissions a month apart may not train on the same libraries. Nothing in that file, or in the tree listing, states a command to reproduce a leaderboard run, which is the gap a new participant hits first.

Editorial conclusion

Parameter Golf fits a researcher who already has a training harness and wants to test a specific idea against a fixed parameter and time budget, and it fits nobody who wants a model to use in production. The top of the board is crowded and the gap between entries is small, so read the summary column rather than the score column before deciding an approach is settled. The repository ships scripts and data, not a package, so you clone it and run the training entry points yourself, and the absent release history means there is nothing to install from.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. openai/parameter-golf on GitHub
4. README