tiny-vllm: a C++ and CUDA inference engine you build yourself, one kernel at a time
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
At a glance
- What is it?
- jmaczan/tiny-vllm is a teaching repository that pairs a working Llama 3.2 1B inference server with a written course covering CUDA kernels, KV cache, continuous batching and PagedAttention. It is aimed at people who want to understand inference internals rather than deploy a server.
- Who is it for?
- Adopt tiny-vllm if you are learning GPU inference internals or teaching a course on them; the repository is explicit that it is a learning tool and a teaching resource, and the course syllabus is the actual product. Do not adopt it as a production serving layer for a model other than Llama 3.2 1B Instruct, because the README lists exactly one supported checkpoint and no release has been cut.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap tiny-vllm fills: inference engines are taught as black boxes
Most engineers meet LLM serving through a Python API. You install a package, point it at a weights directory, and tokens come out. Everything between the HTTP request and the sampled token stays hidden, which is fine until you need to change it. The README frames the problem directly: a model file is "a lot of numbers", the architecture is "just a plan, a blueprint", and neither can run on its own. Something has to turn the blueprint into executable code and load the weights at startup. tiny-vllm is that something, written from scratch in C++ and CUDA, with the construction process documented as a course.
The intended reader is stated in the repository: someone on a learning path, or a lecturer who wants a teaching resource for a university course. The author also says the material derives the ideas and maths from scratch and that mistakes will be made along the way. That framing matters for evaluation. This is not a project that competes on throughput benchmarks against established servers. It competes on whether you finish it understanding why a KV cache exists and how a paged one differs from a contiguous one.
What the engine actually implements, from Safetensors to PagedAttention
The README gives a checklist of completed components, and it is unusually specific. Model loading from Safetensors for Llama 3.2 1B Instruct. A full forward pass covering both prefill and decode. All computation in CUDA kernels. KV cache. Static batching, then continuous batching. Online softmax in a FlashAttention-like form, with a link to the University of Washington CSE599m lecture notes. PagedAttention, with a link to the paper.
That list is the architecture in miniature. Prefill and decode are treated as separate regimes, which is why the course has a dedicated section on the distinction before it explains why the KV cache exists. The batching progression is ordered: static first, continuous second, because the second only makes sense once you have seen the waste in the first. The attention progression is ordered the same way: plain attention, then online softmax, then paging. GQA gets its own section, which follows from the Llama architecture rather than from a general interest in attention variants.
The course table of contents is long and granular. It runs from floating-point representation and why bfloat16 is used, through GPU and CPU memory, tokenization, embeddings, RMSNorm with parallel reduction, RoPE, residual connections, cublasGemmEx, a column-major to row-major transposition trick, SiLU, softmax, causal mask, argmax, the feed forward network and buffer reuse, and ends in the paged KV cache and its CUDA kernel. A reader can infer the dependency order of the implementation from that list alone.
Getting it running: what the README does and does not tell you
This is where the material is thin, and it is worth being blunt about it. The README contains no build commands, no CMake invocation, no environment variables, no server flags and no configuration keys. It states that the repository holds two things, the full source of the inference server and the course, and it states that the model is loaded from Safetensors as Llama 3.2 1B Instruct. Beyond that, the operational details are not in the supplied text.
So the honest answer to "how do I run it" is: read the source tree and the course sections for the build and launch steps, because the README does not provide them. What you can plan for from the README's own content is the dependency surface. You need a CUDA toolchain, since the checklist says all computation is done with CUDA kernels. You need cuBLAS, because cublasGemmEx has its own section and the transposition trick exists to feed it. You need the Safetensors weights for that one checkpoint on local disk. If any of those three are missing, the checklist items cannot execute regardless of how the code is organised.
The absence of a release is consistent with this. The repository has no releases retrieved, and the last push is dated 2026-08-23. Treat the build as something you assemble from the source and the course, not something you install.
The course structure is the real interface, and it constrains who benefits
A repository that ships both an engine and a syllabus has two audiences, and they want different things. The engine audience wants a runnable artifact. The course audience wants an ordered explanation. tiny-vllm is optimised for the second, and the ordering is deliberate in ways that a standalone codebase would not be.
Consider buffer reuse. It appears as a course section between the feed forward network and static batching. In a production engine, buffer reuse is an implementation detail you would never document as a chapter. Here it is a chapter because the reader is expected to have just allocated activation buffers for the FFN and to be about to allocate per-sequence buffers for batching. The placement teaches the tradeoff: memory you can share versus memory you must duplicate per request. The same logic applies to the column-major to row-major transposition trick sitting immediately after cublasGemmEx. Those two sections only make sense adjacent to each other.
This is a genuine strength for the teaching use case and a genuine cost for the adoption use case. If you want to lift the PagedAttention kernel into another project, you will be reading course prose to find the assumptions the kernel makes about layout and block size. The README does not claim the code is organised for reuse, and nothing in it suggests a stable internal API.
Where tiny-vllm is the wrong tool
The clearest limitation is model coverage. The README names one checkpoint, Llama 3.2 1B Instruct, and describes loading it from Safetensors. There is no statement about a model registry, a config abstraction, or support for other architectures. If your workload runs a different model family, or a mixture of models, the loading path and the kernel assumptions are not documented as general. The GQA section exists because Llama uses grouped-query attention, which is a hint that attention head configuration is baked into the implementation rather than parameterised away.
The second limitation is that this is a course artifact. The author says the repository is a learning tool and a teaching resource, and says mistakes will be made. That is an honest description of a project whose value is in the reasoning, not in hardened operational behaviour. There is no release, so there is no versioned artifact to pin. There is no homepage. Anyone evaluating it for a serving deployment is evaluating a moving teaching repository.
The third is the dependency on CUDA specifically. The README's argument for C++ and CUDA is hardware efficiency: LLMs are mostly matrix multiplication, which reduces to dot products of vectors, and a GPU beats a CPU when the math volume is large. That argument also means there is no CPU fallback path described. A machine without a supported GPU cannot run the engine as documented.
How it differs from vLLM, and why that difference is the point
The README positions tiny-vllm as "a younger and smaller sibling of vLLM" and links to the vLLM project. The two share vocabulary: continuous batching, PagedAttention, a paged KV cache, online softmax. They do not share a goal. vLLM is a serving system you deploy. tiny-vllm is a reconstruction of the ideas behind such a system, written so that the reader can follow each step.
That difference shows up in what each project spends its surface area on. A production server needs model coverage, quantisation options, a scheduler that survives adversarial traffic, metrics, and a stable HTTP contract. tiny-vllm's checklist stops at PagedAttention and its kernel. The course spends sections on bfloat16 representation and on why the KV cache exists at all, which a production project would treat as assumed knowledge.
The practical consequence: if you need to serve traffic today, vLLM is the tool the README itself points at as the larger sibling. If you need to understand why vLLM's scheduler and memory manager are shaped the way they are, tiny-vllm walks you through building a smaller version of the same machinery, including the online softmax formulation that makes FlashAttention-style attention possible without materialising the full score matrix. Reading the tiny-vllm PagedAttention section before reading the vLLM source is a reasonable order of operations, and it is the order the README implies.
Licence, maintenance and what upgrading costs you
The licence is Apache-2.0, which is a permissive licence with an explicit patent grant and requires that you preserve notices and state changes when you redistribute. That is the standard reading of the identifier; it is not legal advice, and if you plan to ship a derivative you should read the licence text yourself rather than take a summary from a review.
On maintenance, the material supports only limited claims. The repository is not archived, the default branch is main, and the last push is dated 2026-08-23. No releases have been retrieved, so there is no changelog to read and no version number to pin against. For a course repository this is close to normal: the artifact is the explanation, and the explanation does not need semantic versioning to stay useful. For anyone treating it as a dependency, it means upgrades are a git pull and a re-read, not a version bump with release notes.
The cost that matters most is not upgrade cost but reading cost. The engine and the course are the same artifact, so extracting a component means extracting it from prose that was written to explain it in sequence. Budget for that. The payoff, if the course delivers on its stated intent of deriving the maths from scratch, is that the PagedAttention kernel and the continuous batching loop stop being opaque.
Editorial conclusion
Adopt tiny-vllm if you are learning GPU inference internals or teaching a course on them; the repository is explicit that it is a learning tool and a teaching resource, and the course syllabus is the actual product. Do not adopt it as a production serving layer for a model other than Llama 3.2 1B Instruct, because the README lists exactly one supported checkpoint and no release has been cut. Verify first that your CUDA toolchain and GPU can build the kernels, that you have the Safetensors weights for that specific checkpoint on disk, and that the PagedAttention section of the course matches the kernel source you intend to read, since those two are what the project's performance story rests on.
Community notes