llms-from-scratch: a ten-notebook path through GPT internals in PyTorch
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
At a glance
- What is it?
- This MIT-licensed repository walks from tokenization to a working MiniGPT across ten Jupyter notebooks, one component at a time. It is a study path for readers who want to implement attention themselves, not a training framework or a model you can deploy.
- Who is it for?
- Adopt this if you want to write the attention math yourself and need a notebook per concept to do it in. Do not adopt it if you need a pretrained checkpoint, a training loop that scales past a single GPU, or a fine-tuning pipeline, because the material covers architecture and stops there.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 99 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap this repository fills: architecture you type yourself
Most introductions to large language models hand you a library call. You load a pretrained checkpoint, pass a prompt, and get text back, with the transformer itself sitting behind an abstraction. That is the right way to ship a product. It is a poor way to learn why the model behaves the way it does. This repository takes the other route. Its stated goal is to build a ChatGPT-like LLM in PyTorch and to break the architecture into simple parts, explaining each one step by step. The audience is the engineer or student who wants to implement multi-head attention, layer normalization and residual connections rather than import them. The README's own framing is a bird's eye view of the Generative Pretrained Transformer architecture, followed by ten notebooks that descend into the individual pieces. The scope is deliberately narrow: architecture, not training data pipelines, not evaluation harnesses, not inference serving.
How the ten notebooks fit together
The repository is organized as a linear sequence, and the README's contents list preserves that order: tokenization, token embeddings, positional embeddings, self-attention, masked multi-head attention, feedforward networks, residual connections, layer normalization, the transformer block, and finally MiniGPT. Each stage has its own file under notebooks/, from 01_tokenization.ipynb through 10_mini_gpt.ipynb. The dependency chain is real. Masked multi-head attention assumes you have already built the single-head version in 04, and the transformer block in 09 assembles the attention and feedforward pieces you wrote in 05 and 06. The final notebook is where the stack becomes a model rather than a collection of functions. Because the deliverable is a notebook series, the artifact you produce is a MiniGPT you can run forward, not a downloadable model. The README does not describe a training corpus, a tokenizer vocabulary file, or a checkpoint release, so the working assumption is that you supply your own data if you want to train anything.
Tokenization and the two embedding layers
The README's tokenization section uses the sentence Every moment is a beginning and shows it split into the tokens Every, moment, is, a, beginning, after which each unique token receives a numerical ID. It is explicit that the ID itself carries no meaning: the README notes that the ID 15745 for Every does not contain information about how the token is used in language. That is the motivation for token embeddings, described as vectors that describe a token's characteristics. Positional embeddings are then added on top of the token embeddings. The README justifies them with a minimal pair: The dog jumps on the cat and The cat jumps on the dog contain the same words but mean different things, and token IDs plus token embeddings alone say nothing about order. The stated mechanism is addition, a position vector summed into the token vector, which is the standard approach and the one the notebooks implement. Note that the README treats tokenization conceptually. It shows a whitespace-style split for illustration but does not commit to a specific tokenizer algorithm or vocabulary size, so if you need byte-pair encoding you will be looking at the notebook, not the README, to see what was actually chosen.
Self-attention as the README presents it
The attention section is the most detailed part of the README, and it gives the formula directly: attention is the softmax of QK transpose divided by the square root of d_k, multiplied by V. The explanation proceeds in numbered steps. First, three vectors, Queries, Keys and Values, are derived from the input embeddings. Second, the dot product between all queries and keys measures how well they match, producing the attention scores. Third, those scores are scaled by the square root of the key dimension, which the README says keeps values stable during training. The motivating example is the same sentence, Every moment is a beginning, with the claim that to understand beginning the model attends to words like moment and Every. This is the standard scaled dot-product formulation, and presenting it as a numbered derivation rather than a single code block is the repository's main pedagogical choice. What the README does not cover is the cost side: nothing in the supplied material discusses the quadratic growth of attention with sequence length, memory-efficient attention kernels, or KV caching at inference. If you are reading this to understand why long-context inference is expensive, you will not find that here.
Getting the notebooks running
Installation is a single command from the repository root: pip install -r requirements.txt. The README adds one caveat worth heeding, that if you are installing torch with CUDA support you should use the correct installation command from PyTorch's official website, because some versions require a specific installation method. That is the whole setup story in the supplied material. There is no environment file, no container definition, no Makefile, and no mention of a Python version floor. The notebooks are under notebooks/ with the naming pattern shown in the README's table, so the entry point is whichever numbered file matches the concept you want. Expect to run them interactively rather than as a script, since Jupyter Notebook is the primary language of the repository. If you are working on a machine without a GPU, the material does not state whether the notebooks are sized to run on CPU, and that is something to check in the first notebook before you plan a longer session.
Where this repository stops
The clearest limitation is scope. The README's contents list ends at MiniGPT. There is no section on pretraining at scale, dataset curation, distributed training, mixed precision, gradient checkpointing, evaluation, or alignment. A reader who finishes notebook 10 has a forward-pass implementation of a small GPT and no path within this repository to a model that generates useful text. That is not a defect in a teaching resource, but it is the boundary you need to know before committing time. A second limitation is the format itself. Notebooks carry state, so cells can be run out of order and produce results that the written narrative does not predict. The README's prose is careful about the conceptual order but says nothing about execution order within a file. Third, the material is version-sensitive in a way the README does not address: PyTorch APIs for attention and normalization have changed across releases, and the README only warns about the CUDA install command, not about API drift in the notebook code. This is the wrong tool if your goal is to fine-tune an existing model, to serve inference, or to compare architectures empirically.
How it differs from using a transformer library
The obvious alternative is to use a framework such as Hugging Face Transformers or PyTorch's own nn.Transformer modules, where attention, layer normalization and the encoder or decoder block are provided and tested. The difference is not quality, it is what you end up knowing. With a library you configure; with this repository you derive. The README's step-by-step treatment of Q, K and V, and its insistence on building masked multi-head attention in notebook 05 after single-head attention in notebook 04, is the opposite of an abstraction layer. The trade-off is real in both directions. Library code is faster to deploy, has more users exercising edge cases, and comes with pretrained weights. This repository gives you no weights, no tokenizer artifact and no inference server. What it gives you is the ability to read a transformer implementation and recognize each block, which is a different skill from calling one. If you already understand attention at the level of the formula, the marginal value here drops sharply.
Licence, maintenance and what to verify
The repository is MIT licensed, which is permissive: you can reuse the notebook code in your own work, including commercially, provided the licence notice is preserved. That is a statement about the licence text, not legal advice, and if you plan to redistribute the code inside a product you should read the LICENSE file in the repository yourself. On maintenance, the supplied material shows a last push of 2026-06-08 and reports no releases, so there is no versioned artifact to pin and no changelog to consult. That means the notebooks are the only source of truth, and any dependency drift lands on you. Before adopting it as a study path, verify three things in the repository itself: whether the notebooks still execute against a current PyTorch, what tokenizer the code actually uses in 01_tokenization.ipynb given the README's conceptual treatment, and whether 10_mini_gpt.ipynb produces a runnable forward pass or stops at the class definitions. Those three checks will tell you more about fit than the README can.
Editorial conclusion
Adopt this if you want to write the attention math yourself and need a notebook per concept to do it in. Do not adopt it if you need a pretrained checkpoint, a training loop that scales past a single GPU, or a fine-tuning pipeline, because the material covers architecture and stops there. Before you invest time, open 04_self_attention_mechanism.ipynb and 10_mini_gpt.ipynb and check whether the code you see matches the depth you need, since the README alone will not tell you.
Community notes