One BPE merge by hand, then a 64M model on someone else's GPU
A hands-on course for building modern LLMs from scratch in PyTorch, with 26 runnable Jupyter Notebooks covering tokenizers, attention, MoE, RLHF, inference, evaluation, and distillation.
At a glance
- What is it?
- modern-llm-notebook is a PyTorch course delivered as 26 runnable notebooks rather than a library, and its method is to make you watch each mechanism happen: follow a BPE merge by hand, trace an attention score, observe the first gradient update. The Chinese notebooks are the source edition with an English mirror behind them, the training spine is a 64M-class DotLM Dense run with an MI300X report, and requirements.txt lists tiktoken twice with two different floors.
- Who is it for?
- modern-llm-notebook fits a reader who wants to understand a mechanism rather than call an API, and who is willing to read the Chinese edition when the English mirror lags behind it. It does not fit someone who wants a maintained library or a versioned release, since the project has no GitHub releases and describes itself as under active development.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 46 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- walkinglabs/modern-llm-notebook288 ★ · +103
| Repository | 2026-08-27 | 2026-11-15 |
|---|---|---|
| walkinglabs/modern-llm-notebook | 185 | 288 |
103 Stars in the last 80 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
Twenty-six notebooks, and the description names the topics
This is a course rather than a package, and the repository description gives the syllabus in one line: 26 runnable Jupyter Notebooks covering tokenizers, attention, MoE, RLHF, inference, evaluation, and distillation. The stated main path is narrower and more concrete. Train a BPE tokenizer, prepare real data, pretrain a 64M-class model, continue with supervised fine-tuning, and only then explore MoE, tool calling, post-training, and inference. That ordering is the argument the project makes about how these topics should be learned, and the file naming follows it, since the first notebook lives at notebooks-en/part1-foundation/01-tokenizer-basics.ipynb with a part directory and a two-digit ordinal. Two directories sit side by side at the root, notebooks/ and notebooks-en/, which is the first sign that this is a bilingual publication with a build step rather than a single source tree.
The Chinese edition is the source, not the translation
A note near the top of the README settles a question that would otherwise cost you an afternoon: the Chinese course is the source edition and the English mirror is being updated alongside it. The project describes itself as under active development and asks for corrections, suggestions, and focused pull requests rather than new material. Practically, that means an issue or a pull request against the English notebooks may be answering a question the Chinese notebooks already answered, and that the two can disagree for a while without either being wrong. The bilingual web reader is the mitigation, carrying a language switch so the same chapter can be read in either edition, and the news for 2026-05 records when the bilingual notebooks and the online reader first shipped, covering the foundations from tokenization and Transformers through to Mini-GPT.
tiktoken appears twice in requirements.txt with two floors
The dependency file is short and flat, and it has a defect worth knowing about before you build an environment from it. The list runs torch at 2.0.0 or newer, torchinfo, numpy, matplotlib, Pillow, scikit-learn, tiktoken at 0.7.0 or newer, transformers, datasets, accelerate, modelscope at 1.9.0 or newer, then tiktoken again at 0.5.0 or newer, followed by jupyter and ipykernel. Two entries for the same package with different minimums means a resolver sees both and takes the stricter, so the second line is dead weight rather than a loosening. Everything else is a lower bound with no ceiling anywhere in the file, and there is no dev, lint, or formatting entry at all. modelscope is also a hard requirement even for a reader who never opens the ModelScope notebook, which is a choice worth questioning if you are installing locally.
The npm package exists to run the web reader
There is a package.json at the root of a notebook course, and it is not for the notebooks. The package is named modern-llm-notebook, marked private, versioned 1.0.0, and described as Jupyter Notebooks for modern LLMs with a React web interface. Every script delegates into a subdirectory: postinstall runs an install with a web prefix, and dev, build, and preview all pass through the same prefix. So the entire Node toolchain exists to build the HTML reader. Alongside it sits myst.yml at the root, which is the MyST Markdown configuration the reader consumes, and a web/ directory holding the React application. If you clone the repository to read the notebooks, none of that is needed; if you clone it to work on the site, the Python requirements are the irrelevant half. The repository also carries a package-lock.json, so the Node side at least is reproducible.
Four news entries in five months, and one reorganisation
The news section is a dated changelog, and it is dense. In 2026-05 the project launched bilingual notebooks and the online reader covering the foundations. In 2026-06 it expanded MLA, distributed training, quantization, tool calling, and systems appendices, with runnable exercises and saved experiment outputs. In 2026-08 it reorganised the course into four parts, rebuilt the inference labs around decoding, acceleration, quantization, and deployment, and expanded post-training and diffusion language model topics. In 2026-09 it added from-scratch training labs covering BPE tokenization, data cleaning, 64M-class DotLM pretraining, SFT, and evaluation, with reproduction scripts and experiment reports. The August reorganisation is the one to notice, because a course that reshuffles into four parts mid-stream will have moved pages out from under any link you saved. The last push is dated September 14, 2026, and there are no GitHub releases at all.
The 64M Dense baseline is the spine, and MI300X is the hardware
The concrete training work described is DotLM at 64M-class Dense scale, following the pretraining then SFT scripts, and the project points at an existing experiment report from an AMD MI300X. Everything else in the topic list hangs off that baseline. The diagram in the training path section shows a main route from raw text to a trained model, with a lower strip for the topics to explore once the Dense baseline is done, and the topic list itself runs RoPE, GQA, MLA, MoE, LoRA, tool calling, preference alignment, distillation, quantization, and inference systems. The data side is treated as part of the curriculum rather than an afterthought: where corpora come from, how Data-Juicer cleans them, and how synthetic data and data mixtures work. The project is explicit that the notebooks explain mechanisms while the scripts and reports document the larger runs. The method behind the notebooks is described in four verbs, intuition, hand calculation, implementation, and observation, and the stated prerequisite is only basic Python and matrix operations, which is a deliberately short list for a course whose spine ends at a 64M-parameter Dense model.
Three ways in, and the quotas belong to the platforms
There is no local install command in the README, and that is deliberate. The lowest-effort route is the published reader, which provides partner launch buttons at the top of each chapter so you can start without setting up anything. The second route is opening the first English notebook directly in Google Colab. The third is a partner platform: ModelScope for opening and running a notebook online, an AMD Radeon Cloud template for running on a GPU, and Colab again with a GPU selected when the platform provides one. The project is careful about what it does not promise, stating that sign-in requirements, GPU availability, and usage quotas follow each platform's current rules. That sentence matters more than it looks, because a course whose main path trains a 64M-class model is only as available as the free tier of whichever host you pick.
A dated run report and a screenshot sit in the root
The root directory is where this project stops looking tidy. Alongside the expected files there is NOTEBOOK_RUN_REPORT_20260630.md, a report whose date is encoded in its filename, and verify-fix.png, a screenshot committed at the top level rather than in assets/. There is also OUTLINE.md and intro.md as planning documents, karpathy_models.py as a reference implementation file, llm_train/ for the training scripts, scripts/ for tooling, external/ paired with a .gitmodules file for submodules, and assets/ for the images the README embeds. Editor and agent configuration is present too, with .claude/, AGENTS.md, CLAUDE.md, and CONTRIBUTING.md alongside a .github/ workflow named quality.yml. One inconsistency to note: repository metadata reports no recognised licence while a LICENSE file sits at the root, so check the file rather than the metadata field.
Editorial conclusion
modern-llm-notebook fits a reader who wants to understand a mechanism rather than call an API, and who is willing to read the Chinese edition when the English mirror lags behind it. It does not fit someone who wants a maintained library or a versioned release, since the project has no GitHub releases and describes itself as under active development. Before you start, install from requirements.txt and expect to resolve the duplicated tiktoken entry yourself, then decide whether you have a GPU or plan to use Colab, ModelScope, or the AMD Radeon Cloud template.
Frequently asked questions
What is the modern-llm-notebook course?
A hands-on course for building modern LLMs from scratch in PyTorch, delivered as 26 runnable Jupyter Notebooks covering tokenizers, attention, MoE, RLHF, inference, evaluation, and distillation. The main path trains a BPE tokenizer, prepares real data, pretrains a 64M-class model, and continues with SFT.
Do I need a local environment to try modern-llm-notebook?
No. The published online reader carries partner launch buttons at the top of each chapter, and the first English notebook opens directly in Google Colab, on ModelScope, or through an AMD Radeon Cloud template. Sign-in requirements, GPU availability, and usage quotas follow each platform's current rules.
Which language edition of modern-llm-notebook should I read?
The Chinese notebooks are the source edition, with an English mirror and a bilingual web reader, and the project says the English mirror is being updated alongside the Chinese one. Corrections, suggestions, and focused pull requests are welcome.
What are the Python dependencies for modern-llm-notebook?
requirements.txt lists torch, torchinfo, numpy, matplotlib, Pillow, scikit-learn, tiktoken, transformers, datasets, accelerate, modelscope, jupyter, and ipykernel, all as lower bounds with no ceilings. tiktoken appears twice, once at 0.7.0 or newer and again at 0.5.0 or newer.
What hardware do the modern-llm-notebook training experiments use?
The project points at DotLM's 64M-class Dense pretraining and SFT scripts and at an existing AMD MI300X experiment report. One of the partner platforms is an AMD Radeon Cloud template for running the notebooks on a GPU, and Colab and ModelScope are the other two.
How actively is modern-llm-notebook maintained?
It describes itself as under active development, with dated news entries for 2026-05, 2026-06, 2026-08, and 2026-09 covering the bilingual launch, appendix expansions, a four-part reorganisation, and the from-scratch training labs. The last push is dated September 14, 2026, and there are no GitHub releases.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/walkinglabs-modern-llm-notebook)Star history chart for your README
[](https://hysenlabs.com/projects/walkinglabs-modern-llm-notebook)Was this useful?