MinivLLM: A Self-Contained vLLM Reimplementation With Attention Benchmarks

The Problem MinivLLM Targets: Attention Code You Can Actually Read

Production inference engines keep their attention kernels behind layers of abstraction. Reading vLLM end to end means moving between Python scheduling code, C++ extensions, and CUDA kernels, with build systems in between. MinivLLM takes the opposite route. The README describes it as a custom implementation of the vLLM inference engine, based on Nano-vLLM, but with self-contained paged attention and flash attention. Self-contained is the operative word: the attention logic lives in the same Python package as the scheduler and the model layers, so a reader can follow a token from prompt to sampled output without leaving the repository.

The intended audience is narrow and specific. The README points newcomers to a companion document, HowToApproachvLLM.md, described as a step-by-step implementation guide covering layers, models, paged attention, CUDA graphs, and scheduling. That framing tells you the project is written for people who want to understand how a vLLM-style engine is assembled, or who want to benchmark attention variants against each other on their own hardware. It is not written for someone who needs an endpoint that serves concurrent users today.

Engine Layout: Scheduler, Block Manager, Runner, and Where the Attention Lives

The repository structure in the README maps the engine into four cooperating pieces under src/myvllm/engine/. Sequence definition handles input prompts. Block management handles KV cache memory on the GPU. The scheduler performs iteration-based scheduling of sequences. The runner implements the actual prefilling and decoding passes, and the engine exposes the generation API. Model layers sit in src/myvllm/layers/ (activation, attention, embeddings), with model implementations in src/myvllm/models/ and shared helpers plus inference context management in src/myvllm/utils/.

That split is the standard paged-attention data flow. Prompts become sequences, the scheduler decides which sequences run in a given iteration, the block manager allocates and frees KV cache blocks for those sequences, and the runner executes attention against those blocks. The two attention strategies the project implements are used at different points in that flow: the README states that flash attention is benchmarked in prefilling time and paged attention in decoding time. Prefilling processes the whole prompt at once, so the online softmax approach with O(N) memory applies. Decoding produces one token at a time, so the kernel gathers from the paged KV cache instead. Keeping those two paths separate is what makes the benchmark scripts meaningful rather than decorative.

Running It: uv, Python 3.11, and a CUDA GPU

The Quickstart is four commands. Install uv with the shell installer from astral.sh, then run uv sync to resolve dependencies, then uv run python main.py for the inference demo, uv run python benchmark_prefilling.py, and uv run python benchmark_decoding.py. Dependencies are managed by uv and listed as transformers, torch, and xxhash.

The constraints are stated plainly in the README. Python must be at least 3.11 and below 3.12, which is a single minor version, not a range. A CUDA-capable GPU is required. There is no CPU path documented. For multi-GPU, the README says to change world_size to a value greater than 1 in the config in main.py; it does not name a config file or show the surrounding keys, so expect to read main.py to find the exact location. No environment variables, no YAML, and no CLI flags are documented.

What main.py actually does, per the README: it creates a small version of Qwen3 with random initialization, builds 60 chat prompts (2 base prompts repeated 30 times each), pushes them through the engine with batch processing, and generates up to 256 tokens per prompt with temperature sampling. The repeated-prompt construction is deliberate. Batched generation with identical prefixes is exactly the case where KV cache reuse and block sharing show their value, so the demo is shaped to exercise the machinery rather than to produce interesting text.

What the Two Benchmark Scripts Compare

benchmark_prefilling.py compares three attention implementations during prompt processing. PyTorch standard materializes the full attention matrix, which the README labels O(N^2) memory. Naive Triton is also O(N^2) and is limited by shared memory constraints to prompts of 128 tokens or fewer, a hard ceiling worth noting because it makes that variant unusable for long prompts. Flash attention uses the online softmax algorithm to process attention in blocks, which the README describes as O(N) memory.

benchmark_decoding.py compares three implementations during token-by-token generation. Naive PyTorch is a simple loop over the paged KV cache. Optimized PyTorch is a vectorized version using batch gathering and masking. The third is a custom Triton kernel written for paged attention decode.

The comparison is framed around memory complexity and kernel strategy, which is the honest axis for this kind of code. The README does not publish measured throughput, latency, or memory figures for any of the six variants, and it does not state what hardware the numbers (if any) were produced on. You run the scripts and get your own results on your own GPU. Treat any expectation about which variant wins as something to establish locally, not something the repository asserts.

Limits: No Releases, a Random-Weight Demo, and a Pinned Python

The most concrete limitation is that no releases were retrieved for this repository. Installation means cloning and running uv sync against the main branch at whatever commit is current. There is no version to pin, no changelog to read, and no upgrade path to reason about. If the engine API changes, you find out by running the code.

The demo model is randomly initialized. main.py builds a small Qwen3 and samples from it, so the output text is not meaningful and cannot be used to sanity-check generation quality. What you can verify from that run is that the pipeline completes: prompts are scheduled, blocks are allocated, tokens are produced. That is a smoke test, not an evaluation.

The Python pin is a real constraint. Requiring 3.11 and excluding 3.12 means MinivLLM cannot share an environment with projects that have moved to newer interpreters. Combined with torch and a CUDA GPU, that makes the dependency footprint heavier than the README's three-package list suggests; uv sync has to resolve a CUDA-matched torch build, and that is where most of the install time goes.

Finally, the project is a reimplementation, not a fork of vLLM with a compatibility layer. Nothing in the README claims API parity with vLLM, and there is no mention of OpenAI-compatible endpoints, continuous batching under load, or multi-tenant serving. Judging it against a production engine would be a category error.

MinivLLM Versus Nano-vLLM: What the Rewrite Buys You

The README names Nano-vLLM as the base and states that MinivLLM differs by having self-contained paged attention and flash attention implementations. That is the whole delta, and it is a meaningful one for a specific reader. If you inherit Nano-vLLM and want to modify the attention path, you are editing someone else's kernel code and reasoning about it at a distance. MinivLLM puts that code in the same package as the layers and the engine, so the prefilling and decoding attention variants are inspectable and swappable alongside the scheduler.

If you want a maintained, feature-complete engine, neither project is the right comparison. vLLM itself is the alternative, and the difference in approach is structural: vLLM optimizes for serving many concurrent requests with a large kernel and build surface, while MinivLLM optimizes for a reader being able to trace the whole path in Python. The benchmark scripts exist precisely because the second goal benefits from being able to compare a naive implementation against an optimized one on the same machine. If you need throughput, use vLLM. If you need to understand why vLLM's decode kernel looks the way it does, MinivLLM's decoding benchmark puts the naive loop, the vectorized version, and the Triton kernel side by side.

Licence, Maintenance, and What Upgrading Costs

The repository is Apache-2.0. That is a permissive licence with an explicit patent grant and a requirement to preserve notices and state changes when you redistribute. It is compatible with commercial use. This is a description of the licence text, not legal advice; if you plan to ship derivative code, have your own counsel review the NOTICE and attribution requirements.

Maintenance signals are thin. The last push recorded is 2026-08-29, and no releases were retrieved, so there is no versioned artifact to track. Upgrade cost therefore has two components. First, there is no upgrade: you re-clone or pull main and re-run uv sync, and any breakage surfaces at that point. Second, the Python 3.11 pin means that when you eventually need a newer interpreter for other dependencies, MinivLLM is the package that blocks the move until it is relaxed. Planning around a project with no releases means budgeting for a pull-and-test cycle rather than a dependency bump.

Editorial conclusion

Adopt MinivLLM if you want to read a working paged attention and flash attention implementation in Python and run the two benchmark scripts to see the memory behaviour of each variant. Do not adopt it as a serving layer for real traffic: the demo uses a randomly initialized small Qwen3, there is no release, and no throughput or latency numbers are published. Before committing time, verify that your GPU and driver satisfy the CUDA requirement, that Python is pinned to 3.11 (the README allows no other version), and that uv sync resolves torch against your local CUDA build.

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Wenyueh/MinivLLM on GitHub