tiny-llm: A Hands-On Course for Building an LLM Inference Stack on Apple Silicon

What tiny-llm Actually Teaches

tiny-llm is a course, not a library. It targets systems engineers who know how to write code but have not connected the mathematics of transformers to the realities of memory bandwidth, kernel occupancy, and request scheduling. The README positions it as an LLM-serving counterpart to CMU's Needle project, which is a useful comparison: both ask you to build a small but complete implementation from primitives. The course is built on MLX arrays and the MLX extension runtime, but it deliberately avoids high-level neural-network layers. When a chapter teaches an operator, you implement that operator yourself in Python, C++, or Metal, and MLX serves only as the correctness oracle and performance baseline. That design choice is the core of the project's value: it forces you to touch every layer of the stack, from matmul to paged attention, without the convenience of a framework hiding the details.

The Four-Week Path from Matmul to a Mini vLLM

The learning path is structured into four weeks. Week 1 builds a Qwen3 model directly from mlx.core array operations, covering attention, RoPE, grouped-query attention, RMSNorm, the MLP, sampling, and the autoregressive loop. Week 2 introduces a KV cache and establishes a synchronized MLX baseline, then uses matched benchmarks to decide between optimizations like quantized decode matvec, fused model kernels, tiled prefill, and split-K. Week 3 shifts to serving machinery: continuous batching, chunked admission, and paged KV cache as the canonical layout. Decode attention and FlashAttention are taught to read pages directly, avoiding the need to rebuild dense history each step. Week 4 diverges into building a coding agent, which is a surprising but pragmatic choice. It covers a validated agent loop, workspace inspection, approved edits, checkpointing, and evidence management. The progression is logical: you start with tensor operations, then move to memory management, then to scheduling, and finally to an application that uses the inference engine you built.

Why Qwen3 and Apple Silicon Are the Right Fit

The choice of Qwen3-4B and Apple Silicon is not arbitrary. Apple Silicon provides a unified memory space with direct access to Metal kernels, which lets you inspect the entire path on one machine without a CUDA GPU. Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally. The README mentions that Qwen3's grouped-query attention, QK normalization, BF16 activations, and 4-bit weights keep the exercises close to current model-serving practice. This is a meaningful advantage over using a tiny model like GPT-2, where memory traffic is not a bottleneck and optimizations feel academic. The course is built on MLX, which is Apple's array framework, so it will not run on NVIDIA GPUs or even on Apple Silicon with a different framework. If you are on an Intel Mac or a Linux machine, this course is not for you.

Getting Started: Commands and Repository Layout

The README gives concrete commands to verify an existing checkout. You need pdm, which is a Python dependency manager. The commands are: 'pdm install -v' to install dependencies, 'pdm run check-installation' to verify the environment, and 'pdm run test-refsol -- -- -k week_1' to run the reference solution tests for Week 1. The repository has two main packages: 'tiny_llm' where you implement exercises, and 'tiny_llm_ref' which contains the reference solution used by tests and benchmarks. The book is published at skyzh.github.io/tiny-llm, and the README points to an environment setup page. The 'book/src/SUMMARY.md' lists the chapter order. This setup is straightforward for anyone familiar with Python packaging, but note that the course expects you to write Metal kernels, so you need the Xcode command line tools and a Mac that supports Metal. The course is not a pip-installable package; it is a source repository you clone and work through.

The Reality of Course Completion: Roadmap and Audit Status

The course is not fully complete. The roadmap table tracks four dimensions: implementation code, tests, rendered documentation, and an editorial audit by a reviewer named Chi. Week 1 is fully done in all four columns. Week 2 and Week 3 have code, tests, and documentation marked as done, but the audit column shows a construction symbol, meaning the learner-facing material has not been editorially reviewed. Week 4 is publishing one day at a time, with Days 1 through 9 currently available. This is a significant limitation. If you are the kind of learner who relies on polished prose to explain subtle concepts, the later chapters may have rough edges. The audit column is independent of code and test readiness, so even though the code works, the explanations may not be as clear. The README also warns that Day 3 involves sending file contents to a model and running commands, so you should use a disposable workspace without secrets. That is a concrete safety note, not generic advice.

What You Will Not Learn, and Where It Falls Short

The README explicitly says that other topics are not covered, including quantized, but the sentence is truncated. Based on the table, optional chapters include Mixture of Experts and Speculative Decoding, so those are covered but marked optional. The course does not cover distributed inference, multi-GPU setups, or production deployment concerns like fault tolerance or horizontal scaling. It is also not a substitute for reading the vLLM source code: the mini vLLM you build is a pedagogical simplification, not a production system. The biggest gap is that the course assumes you have a Mac with enough unified memory to run Qwen3-4B. The README does not specify minimum memory requirements, but a 4B model with 4-bit weights needs at least 2 GB just for weights, plus KV cache and activations, so realistically you need 8 GB or more. If you have a base M1 Mac with 8 GB, you may struggle with the prefill kernels. Another limitation is that the course is time-boxed to four weeks, which means you will not have time to explore every optimization in depth. The benchmarks are meant to guide choices, but the course does not provide a full performance analysis of each kernel.

Alternatives and How They Differ

The most direct alternative is the real vLLM project, which tiny-llm explicitly mimics in Week 3. vLLM is a production-grade inference engine that implements continuous batching, PagedAttention, and chunked prefill in optimized CUDA kernels. The difference is in approach: vLLM gives you a working system to deploy, while tiny-llm asks you to reimplement those ideas in a small, readable codebase. If you want to learn, tiny-llm is better because you write the kernels. If you want to serve models, vLLM is the tool. Another alternative is the Needle project from CMU, which tiny-llm cites as inspiration, but Needle focuses on deep learning systems more broadly, not specifically on LLM serving. There is also the MLX documentation itself, which provides examples for building models, but it lacks the structured curriculum and reference solutions that tiny-llm offers. The key difference is that tiny-llm is a course with tests and a reference implementation, so you can verify your work, whereas other tutorials are often just code snippets.

Maintenance, Licensing, and Upgrade Cost

The repository is under the Apache-2.0 license, which is permissive for both academic and commercial use. The last push was on September 9, 2026, and the project is not archived, so it is actively maintained. However, because it is a course, the maintenance burden is on the author to keep the exercises aligned with changes in MLX and Qwen model versions. The README includes a CI badge for a main workflow, indicating that tests run automatically. For a learner, the upgrade cost is low: you clone the repository and follow the setup. But if you want to use it as a base for your own project, you will need to adapt the code to your specific model or hardware, which is not trivial because the exercises are tightly coupled to Qwen3 and MLX. The reference solution is included, which is a double-edged sword: it helps you check your work, but it may tempt you to peek when you are stuck. The course's value depends on your discipline to implement before looking at the reference.

Editorial conclusion

Adopt tiny-llm if you are a systems engineer or student who learns best by implementing rather than reading, and you have an Apple Silicon machine with enough memory to run Qwen3-4B. Avoid it if you need a deployable inference server or if you prefer to use high-level MLX layers, since the course explicitly forbids them in exercises. Before starting, verify that your Python environment supports MLX and that you can run the provided check-installation command; also confirm the book's environment setup page matches your macOS version. The course is incomplete for Week 2 and beyond regarding editorial audit, so expect some rough edges in later chapters. If you want a production-grade vLLM, use the real vLLM project instead; tiny-llm is for understanding, not serving at scale.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. skyzh/tiny-llm on GitHub