llm-architecture-from-scratch: one PyTorch reimplementation per model family

The table is the catalogue, and it is the most useful page in the repository

llm-architecture-from-scratch is a collection of PyTorch implementations of language model architectures, and the README is essentially a comparison table plus a work in progress list. The table has seven columns: model, model size, modality, normalization, positional encoding, attention, and whether the model is a mixture of experts.

That structure is the point of the project. Reading the table alone teaches the shape of the field. GPT-2 XL, the oldest entry, is the only one using LayerNorm and learned absolute positional encoding, and it is the only one with plain multi-head self-attention. Everything after it moves to RMSNorm and rotary positional encoding, and stays there for the rest of the list.

The size column ranges from the 1.5B of GPT-2 XL up to the 671B of DeepSeek v3, with 400B for Llama 4 Maverick and 235B A22B for the larger Qwen3. Three entries are listed as text and image: Gemma 3 at 27B, Mistral 3.1 Small at 24B, and nothing else. The modalities column is therefore mostly a wall of text, which is a useful correction to anyone who assumes recent releases are all multimodal.

The repository layout matches the table one for one. Each model has its own top level directory, such as gpt2-xl/, llama3-8b/, deepseekv3/, gemma3/, qwen3_32b/ and xlstm/, alongside glm4_5/, kimi_k2/ and gpt_oss/ for work that is still in progress.

None of the entries cite parameter counts beyond the model size, so treat the sizes as labels for scale rather than as configuration values you can read off the code.

Attention is where the implementations actually diverge

The attention column is the most informative one, because it is where modern architectures stopped sharing a single design. The list covers four distinct variants.

Grouped-Query Attention, causal, appears on the majority of entries: Llama3-8B, Llama 3.2-3B, OlMo 2 7B, Phi-4, DeepSeek v3, Qwen3 and Qwen3 235B A22B. Sliding window combined with GQA appears on Gemma 3 and Mistral 3.1 Small, the two multimodal entries. Llama 4 Maverick is listed simply as GQA. xLSTM is the outlier, listed with mLSTM as its attention, which is a recurrent design rather than a quadratic one.

That distribution is a good summary of what changed. Full multi-head attention with a separate key and value head per query head is expensive at inference because of the memory bandwidth of the key and value cache, and grouped-query attention reduces that by sharing them across groups of query heads. Sliding window attention adds a second restriction on top, bounding how far back each token can attend, which is what lets a multimodal model keep a long image context affordable.

Two mixture-of-experts entries stand out. DeepSeek v3 at 671B and Llama 4 Maverick at 400B are marked as mixture of experts, and the second Qwen3 configuration, 235B A22B, is as well. The naming convention there is the interesting part, since A22B describes the active parameter count rather than the total, which is the convention that lets a mixture-of-experts model be called a 235B model while activating far fewer parameters per token. By contrast the 32B Qwen3 and every other dense entry in the table have no such split, and a reader comparing a dense and an expert model should compare active parameters, not the headline number.

Position encoding variants, including one model that uses none

Positional encoding is the second axis of variation and the table treats it as a first class column rather than burying it. Learned absolute encoding appears only on GPT-2 XL. RoPE, rotary position embeddings, is used by every other implemented model.

The exception that proves the rule is SmolLM3, listed as RoPE + NoPE. NoPE is the no positional encoding variant, and having it appear in the same cell as RoPE tells you the model mixes the two schemes across its layers or its sequence handling rather than picking one. The repository lists SmolLM3 as 3B, a size where attention cost is not yet the binding constraint, which makes a mixed scheme a plausible design rather than a scaling necessity.

xLSTM carries RoPE as well, but pairs it with mLSTM attention, so its positional handling exists inside a recurrent rather than a self-attention block. That is a genuinely different arrangement of the same two ideas, and seeing both in the same repository is useful precisely because the surrounding machinery is otherwise familiar.

The README does not describe the implementation of any of these, nor the base period, the scaling factors, the interpolation method or the exact NoPE placement in SmolLM3. Anyone wanting those details has to read the code, which is the point of the project but also the limit of what the documentation delivers.

The dependency list is one line, and the install story is thin

The project manifest is minimal. The name is llm-architecture-from-scratch, the version is 0.1.0, the description matches the README tagline, and the requires-python is 3.12 or newer. The dependency list has a single entry:

dependencies = [
    "torch>=2.14.0",
]

One dependency is a deliberate statement about the project. There is no transformers, no accelerate, no datasets and no training framework, so whatever is in each directory is hand written PyTorch rather than a configuration of somebody else's model class. That is what makes the implementations worth reading, and it is also what makes them expensive to run at scale.

The repository root has a .python-version file and a uv.lock, which indicates uv as the environment manager in use, but the README does not give a setup command and the lock file's contents are not shown, so the exact environment invocation is left to the reader. There are no GitHub releases, so there is no tagged version to install either.

The practical route is therefore to clone the repository, create a Python 3.12 environment, install torch, and open the directory for the model you care about. Which raises the question of how far each implementation goes, and the README does not answer it. There is no statement about whether the code includes training loops, whether it loads pretrained weights, or whether it is inference only. The examples directory is absent from the top level listing entirely, and the only per-model artefacts visible are the model directories themselves.

Where the repository stops short of what you might expect

The limitations are structural, and they follow from what the project is not.

There are no pretrained checkpoints, so nothing here can be run to produce a completion out of the box. The purpose stated in the description is implementing LLM architectures for deep understanding, and the code is the deliverable. If your goal is to serve a model, this is the wrong repository and you want the original release instead.

There is also no training data pipeline, no tokenizer training, and no evaluation harness in the top level listing. A directory listing at the root shows .gitignore, .python-version, LICENSE, README.md, the model directories, pyproject.toml and uv.lock, with no tests directory and no scripts directory. That means you cannot check a port against a reference implementation from within the repository, and you cannot measure whether your run of it matches the published weights.

The in progress list makes the gap explicit. Kimi K2 at 1T, GLM 4.5 at 355B, GPT OSS 120B, Grok 2.5 at 270B and Qwen3 Next at 80B-A3B are all named as work in progress, so five of the directories exist but are not finished.

A third limit is scale, and it is the one the table makes obvious. A 671B DeepSeek v3 implementation is a set of modules and shapes, not something you can instantiate on a single GPU. Every entry in the table above 30B is a reading exercise in the design, which is a legitimate use, but it is a different use from running the model.

Finally, the absence of a contributions guide, a changelog and a code of conduct in the top level listing means the conventions of each directory are whatever its author left behind.

Against a framework implementation, the difference is the code you have to read

The usual alternative to a repository like this is the model class in a mainstream framework, where a config file and a class name give you the architecture without showing the mechanics. The difference in approach is exactly what the reader is looking for.

A framework implementation is authoritative in the sense that it is what people run. It is also compressed into configuration, and a subtle choice such as how the rotary frequencies are applied, or where the RMSNorm sits relative to the attention, is invisible in a config. Reading a hand written port forces those choices into the open as ordinary Python lines.

The comparison this repository is best suited to is between architectures, not between implementations of the same one. Having GPT-2 XL and Qwen3 235B A22B in the same tree means the difference between learned absolute positional encoding and rotary, and between dense attention and expert routing, can be read as a diff rather than reconstructed from two separate papers.

The cost is the obvious one. Hand written code in thirteen directories has no shared test suite, no guarantee of consistent naming, and no promise that each directory was ported to the same standard. A framework gives you a uniform API and a maintained codebase, and if your goal is working output rather than understanding, that is the better trade.

The BSD-3-Clause licence reinforces the practical side. It is permissive, it allows use in commercial and closed products, and it requires only that the copyright notice and licence text travel with copies, which makes the code safe to fork into an internal training stack.

Editorial conclusion

llm-architecture-from-scratch fits an engineer who wants to read working PyTorch for a specific architecture, especially the ones where a design choice such as grouped-query attention, sliding window attention, NoPE, mLSTM or a mixture-of-experts router is easy to find in a real paper and hard to find in a framework. It does not fit a team looking for pretrained weights, a serving stack or a training recipe, because the repository ships implementations and a single torch dependency with no checkpoints and no dataset pipeline. Verify three things before you build on it: that the directory for your model of interest contains the configuration you expect, that the license file covers your use, and that the code is a faithful port rather than a simplification, since the README states an aim of deep understanding rather than numerical parity. The project is BSD-3-Clause licensed, and the last push was on 2026-10-17.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. README
4. YeonwooSung/llm-architecture-from-scratch on GitHub