OpenArch: Hand-Written PyTorch Implementations of Modern LLM Architectures

What OpenArch solves, and who it is actually for

Reading official model code is unpleasant for a specific reason. Production repositories optimize for speed, sharding and backward compatibility, so the architectural decision you are looking for is buried under distributed wrappers. OpenArch takes the opposite position. The README states the goal is clarity and learning: a single readable file per architecture, with structural choices made explicit. Attention type, normalization, layer mix, MoE routing and positional encoding are the axes it wants you to compare side by side.

The intended reader is someone who already knows PyTorch and wants to see how Grouped Query Attention differs from Multihead Latent Attention in code rather than in a diagram. It is also aimed at contributors. The README says the author is actively looking for contributors and lists good first contributions, including picking an unimplemented model from the gallery, writing a README for an existing model, or adding a forward-pass test that loads official weights and matches outputs on a few tokens. That last item is telling: it is listed as a contribution, which means it is not uniformly present yet.

The repository layout and how one model maps to one file

The layout is flat and predictable. A text/ directory holds one folder per model, each with a model.py and a short README.md describing the architectural choices and references used. A multimodal/ directory holds PaliGemma in the same shape. The top level carries CONTRIBUTING.md, LICENSE, README.md, requirements.txt, image/, multimodal/ and text/.

That mapping is the whole design. There is no shared registry, no model hub, no auto-discovery. If you want to know how DeepSeek R1 routes experts, you open text/deepseek_v3/model.py and read it. The README describes implementations as hand-written from the original papers, technical reports, reference config.json files, and writeups by Sebastian Raschka and Machine Learning Mastery. The trade-off is explicit: no abstraction layer means no reuse across models, so shared components get refactored by hand or duplicated. For a reading exercise that is fine. For anything you intend to maintain across twenty models, it is friction.

Installing OpenArch and running a first forward pass

There is no package on PyPI and no install section in the README. The repository ships a requirements.txt listing transformers, torch and numpy, with no pinned versions. Clone the repository and install those three, then open the model file you care about.

git clone https://github.com/anuj0456/OpenArch.git
cd OpenArch
pip install -r requirements.txt

The requirements.txt content is exactly this, with no version pins:

transformers
torch
numpy

After that, the README does not give a runnable entry point, a script name or a CLI. The documented unit of use is the file itself, so the first real step is reading text/llama3/model.py and instantiating the class it defines with a configuration you supply. The README does not document a loader that maps Hugging Face config.json files onto these classes, and it does not document weight conversion. Treat the first session as code reading, not as an inference run, and expect to write the small driver yourself.

Which architectures are usable today and which are under construction

The README marks implementations with a check for forward-pass usability and a construction symbol for work in progress. Usable text models include GPT-2 XL, Llama 2, Llama 3, OLMo 2, DeepSeek R1, Gemma 3, Mistral 3, Llama 4 Maverick, Qwen 3 at both 4B and 30B-A3B, Kimi K2, GLM 4.5 and GPT-OSS. Grok-2.5 is listed as under construction. In multimodal, PaliGemma is usable and Qwen3 is under construction. Dall-e sits under Image, also under construction.

The comparison table is the most useful artifact in the repository. It tells you, for example, that Llama 3 uses Grouped Query Attention while Llama 2 uses Multihead Attention, that OLMo 2 adds QK-Norm on top of RMS Norm, and that Gemma 3 and Mistral 3 pair Grouped Query Attention with a sliding window. DeepSeek R1 and Kimi K2 both use Multihead Latent Attention and Mixture of Experts. GLM 4.5 is listed with Grouped Query Attention and Multi-Token Prediction. The README says the full target list mirrors the 72 architectures in the gallery, so the table is a progress tracker as much as a reference.

Where OpenArch is the wrong tool

The README is direct about this: the goal is not to compete with transformers or other production libraries. Take that seriously. There are no releases, no versioned API, no published benchmarks and no documented support policy. The requirements.txt pins nothing, so a future transformers or torch release can change behaviour under you without any signal from this repository.

The larger models in the table make the point. DeepSeek R1 is listed at 671B, Llama 4 Maverick at 400B, Kimi K2 at 1T. Nothing in the README suggests these files are memory-optimized, sharded or quantized. They are readable implementations of the architecture, which is a different object from a runnable deployment. If your task is serving a model, fine-tuning at scale, or matching upstream logits bit for bit, this is the wrong repository. If your task is understanding why a sliding window changes the attention mask, it is a reasonable one.

How OpenArch differs from transformers and from nanoGPT

The comparison the README itself draws is with transformers. The difference is scope and intent rather than features. transformers maintains a large compatibility surface across hundreds of checkpoints, with configuration classes, tokenizers, generation utilities and device placement handled for you. OpenArch has none of that. It keeps the architectural skeleton and drops the surrounding machinery, which is why a single file can hold a whole model.

The other natural comparison is nanoGPT, which the README does not mention. nanoGPT concentrates on one GPT-style training loop and makes it end to end. OpenArch spreads across many architectures and stops at the model definition. So nanoGPT answers how do I train a small GPT, while OpenArch answers how does Gemma 3 differ from Mistral 3 in attention and normalization. Neither replaces transformers, and picking between them depends on whether you want a training loop or a comparison table.

Licence, maintenance and the cost of keeping up

OpenArch is MIT licensed. That permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. It is a permissive licence, and nothing in the README adds a contributor licence agreement or a second set of terms. This is a description of the licence file, not legal advice; check LICENSE and your own obligations before shipping anything derived from it.

The last push to the default branch was on 2026-09-14, and the repository is not archived. There are no retrieved releases, so there is no version to pin and no changelog to read. Upgrade cost is therefore not a versioning question but a drift question: architectures change, and a hand-written file based on a paper and a config.json will fall behind the model it describes. The README's answer is contributions, including forward-pass tests against official weights. Until those tests exist for a given model, verifying that a file still matches its architecture is manual work you own.

Editorial conclusion

OpenArch is for engineers who want to read attention, normalization and MoE routing choices side by side in plain PyTorch, and for contributors willing to add a model.py from the gallery. It is not for anyone who needs a supported inference or training stack, because the README positions it against transformers rather than beside it. Before adopting it in any workflow, open text/llama3/model.py, confirm the forward pass matches the config.json you care about, and check whether the model you need is still marked under construction.

Frequently asked questions

Official sources

1. anuj0456/OpenArch on GitHub
2. Issues
3. License: MIT
4. README