Mini-SGLang: A 5,000-Line Reference Implementation of an LLM Serving Stack

The gap Mini-SGLang fills between a tutorial and a serving engine

Most explanations of LLM serving stop at a diagram: requests arrive, a scheduler batches them, a KV cache holds attention state, a model runs. The distance between that diagram and a working server is where the interesting decisions live, and it is usually hidden inside tens of thousands of lines of code. Mini-SGLang is an attempt to close that distance by reimplementing SGLang in roughly 5,000 lines of Python, as the README puts it, to demystify the complexities of modern LLM serving systems. The target reader is a researcher or developer who wants to read the scheduler, not call it. The README also describes it as a capable inference engine, so the project does not present itself as a toy. That dual framing is the central tension of the repository: the same code is offered as both a teaching artifact and a runnable server, and the documentation does not draw a line between the two roles.

Radix cache, chunked prefill and overlap scheduling in one scheduler

The feature list in the README names four mechanisms that interact inside the serving loop. Radix cache reuses KV cache for shared prefixes across requests, which matters when many prompts begin with the same system message. Chunked prefill splits long prompts into pieces so peak memory during prefill is lower, at the cost of more scheduling steps. Overlap scheduling hides CPU scheduling overhead behind GPU computation, meaning the scheduler prepares the next batch while the current one is still executing. Tensor parallelism scales the model across GPUs. On top of that, the project integrates FlashAttention and FlashInfer as attention kernels. The README points to docs/structures.md for the design and data flow, and docs/features.md for the full argument list. That is as far as the supplied material goes: the repository layout implies a scheduler module, a cache module and a model runner, but the README does not name the files, so any claim about how the overlap is implemented at the Python level would be guesswork. The ablation switch is concrete, though. Setting MINISGL_DISABLE_OVERLAP_SCHEDULING=1 turns overlap scheduling off, which is how the offline benchmark isolates its effect.

Installing Mini-SGLang on Linux with uv and a matching CUDA Toolkit

The README is explicit that Mini-SGLang supports Linux only, on x86_64 and aarch64, because it depends on Linux-specific CUDA kernels from sgl-kernel and flashinfer. Windows and macOS are not supported; the README suggests WSL2 or Docker instead. Installation from source uses uv. The documented sequence is git clone of the repository, then uv venv --python=3.12, then source .venv/bin/activate, then uv pip install -e . The README recommends Python 3.10 or newer. It also warns that the CUDA kernels are JIT-compiled, so the NVIDIA CUDA Toolkit must be installed and its version must match the driver version, checkable with nvidia-smi. Serving starts with python -m minisgl --model "Qwen/Qwen3-0.6B". Multi-GPU runs add --tp, for example --tp 4 with meta-llama/Llama-3.1-70B-Instruct on port 30000 via --port. Adding --shell opens an interactive terminal chat, where /reset clears history. If HuggingFace downloads fail, --model-source modelscope switches the download source. The Docker path builds with docker build -t minisgl . and runs with docker run --gpus all -p 1919:1919, and the README recommends mounting volumes for the HuggingFace, tvm-ffi and flashinfer caches so subsequent startups skip recompilation.

The benchmark switches tell you what the project wants you to measure

Two benchmark scripts are referenced: benchmark/offline/bench.py and benchmark/online/bench_qwen.py. The offline configuration described in the README is one H200 GPU, Qwen3-0.6B and Qwen3-14B, 256 sequences, with input and output lengths randomly sampled between 100 and 1024 tokens. The online configuration is four H200 GPUs connected by NVLink, Qwen3-32B, replaying the first 1000 requests of the Qwen trace dataset. The online comparison launches Mini-SGLang with --tp 4 --cache naive and SGLang with --disable-radix --decode-attention flashinfer. That --cache naive flag is worth pausing on. The comparison deliberately disables the radix cache on the Mini-SGLang side and the radix cache on the SGLang side, which means the published online numbers are not measuring the prefix-reuse path at all. If your workload has heavy shared prefixes, this benchmark says nothing about it. The README does not state the result values in text; they appear only as linked images, so no throughput or latency figure can be quoted from the supplied material.

Where Mini-SGLang is the wrong tool

The README lists no releases, which means there is no versioned artifact to pin and no changelog describing what changed between states of the code. Installation is from a git clone with uv pip install -e ., so upgrades mean pulling the branch and reinstalling. For a production endpoint that is a real operational cost, and the README does not describe a stability policy or a compatibility guarantee for the CLI flags. The platform restriction is the second hard boundary: Linux only, with CUDA kernels JIT-compiled at first run, so a cold start on a new machine pays a compilation cost that the README addresses by recommending persistent Docker volumes for the tvm-ffi and flashinfer caches. Third, the project is a reimplementation of SGLang, not a superset. Anyone who needs the full feature surface of the upstream server, including whatever options exist beyond the ones named here, should read docs/features.md rather than assume parity. The README does not claim parity, and nothing in the supplied material suggests Mini-SGLang tracks upstream releases.

Choosing between Mini-SGLang and SGLang itself

The honest comparison is not between Mini-SGLang and vLLM or TensorRT-LLM, because the README does not benchmark against them. It is between Mini-SGLang and SGLang, the project it reimplements, and the difference is scope rather than approach. Both use radix caching, tensor parallelism and FlashInfer-style attention kernels; the online benchmark command even runs them side by side on the same model and GPU count. What separates them is the size of the codebase and what that size buys. SGLang carries the accumulated features, edge cases and operational tooling of a project used at scale. Mini-SGLang carries 5,000 lines that a single reader can hold in their head, which is the entire point. If you want to change how the scheduler batches, or add a cache policy, the smaller codebase is the one you can modify in an afternoon. If you want a server that someone else maintains, the smaller codebase is the one with no release tags. That is the trade, and the README's own framing supports it: the stated purpose is transparency, and performance is described as a property of the design rather than the reason to choose it.

Licence, maintenance and what to check before committing

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive baseline, but this is not legal advice and the licence text in the repository is the authority. Maintenance cost is harder to estimate from the supplied material. There are no releases, the last push recorded is 2026-05-17, and the README does not describe a contribution process, a test suite, or a support channel. For a project whose value is readability, that is less alarming than it would be for a serving dependency, because you can read the diff yourself when you upgrade. The concrete checks before adopting are: confirm your CUDA Toolkit version matches your driver, confirm sgl-kernel and flashinfer build on your target GPU architecture, and read docs/features.md and docs/structures.md to see whether the architecture matches the workload you have in mind. If your workload depends on prefix reuse, note that the published online benchmark runs with --cache naive and does not exercise it.

Editorial conclusion

Adopt Mini-SGLang if you are studying or teaching how a modern LLM server schedules prefill and decode, or if you want a small codebase to modify for experiments. Do not adopt it as a production endpoint: the README lists no releases, no stability policy and no upgrade path, and the platform support is Linux only. Verify first that your CUDA Toolkit version matches your driver, that sgl-kernel and flashinfer compile on your target GPU, and that the benchmark configuration in benchmark/offline/bench.py matches your own batch shape before drawing any throughput conclusion.

Official sources

1. Issues
2. License: MIT
3. README
4. sgl-project/mini-sglang on GitHub