ZhiLight: Zhihu's C++ Inference Engine for PCIe GPUs

The PCIe interconnect problem ZhiLight was built around

On PCIe-based GPU nodes, tensor parallelism stalls on all-reduce traffic that NVLink-class machines absorb without noticing. ZhiLight is positioned explicitly around that constraint: the README says it accelerates "Llama and its variants, especially on PCIe-based GPUs." The benchmark environments listed are AD102 PCIe, A800 and H20, all PCIe-attached parts, so the target is the deployment shape most teams can actually rent or buy, not an HGX board. The intended user is an infrastructure engineer serving dense 2B to 110B checkpoints, or a DeepSeek-V3/R1 deployment, who has measured inter-GPU communication as the bottleneck and wants the engine to attack it directly. It is not a research framework. There is no training path here, and the feature list is entirely inference-side: dynamic batch, chunked prefill, prefix cache, quantized kernels.

Dual streams, host all-reduce and the rest of the overlap machinery

The distinguishing mechanism in the README is what the project calls "dual streams": encode and all-reduce overlap. Rather than serializing the reduction of gradients (in inference, the reduction of partial attention and MLP outputs across tensor-parallel ranks) after the compute kernel finishes, ZhiLight schedules the communication on a second stream so it runs alongside encoding work. The README adds that the all-reduce itself can be Int8-quantized "to further reduce all-reduce cost," and that there is a host-side all-reduce path built on SIMD instructions. Those three items are one design argument: when the link is PCIe, you shrink the payload (Int8), overlap it with compute (dual streams), and keep a CPU fallback for the reduction (SIMD host all-reduce). Everything else is the usual serving stack. There is an OpenAI-compatible asynchronous interface the README says is adapted from vLLM, a custom tensor type with unified global memory management, fused kernels for QKV, residual and layernorm, tensor-core fused batch attention for the decode step, and flash attention prefill. Quantization coverage is broad on paper: native INT8, SmoothQuant, FP8, AWQ and GPTQ, with a Marlin kernel for GPTQ. MoE support covers DeepseekV2 MoE and DeepseekV2 MLA. The model list runs from Llama and Llama2 through Mixtral and Qwen2 to DeepSeek-VL2 multimodal checkpoints added in January 2025.

Building the wheel and starting the server

The README gives four concrete commands. To build a wheel with parallel compilation and unit tests disabled: CMAKE_BUILD_PARALLEL_LEVEL=32 TESTING=0 python setup.py bdist_wheel. To build with the Ninja backend: CMAKE_GENERATER="Ninja" python setup.py bdist_wheel (note the spelling of the variable as published, which is what the README shows). To install in place: cd ./ZhiLight && pip install -e . To start the OpenAI-compatible server: python -m zhilight.server.openai.entrypoints.api_server [options]. The README does not enumerate what those options are, which is a gap you will hit immediately if you want to set tensor-parallel size, model path or port from the command line. Dependencies are described as narrow: the CUDA runtime, cuBLAS, NCCL and the packages in requirements.txt, with a Dockerfile under docker/ as the reference. A prebuilt image is published at ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124, matching the v0.4.8 release from December 2024 and CUDA 12.4. The server entrypoint module path is the one thing to verify first, since the package layout under zhilight/server/openai is not otherwise documented in the README.

The benchmark tables do not all point the same way

The performance section is unusually honest for a project README, and it is worth reading closely rather than skimming the bold cells. On DeepSeek-R1 AWQ with eight A800s, ZhiLight shows 0.16 QPS against vLLM's 0.08, with lower TTFT mean and P95, though TPOT mean and P95 are slightly worse (115.97 and 139.99 against 109.86 and 129.98). On MiniCPM-2B-sft-bf16 on a single AD102, all three engines report 1.67 QPS; ZhiLight leads on TTFT and trails SGLang on TPOT P95. On Qwen2-72B-Instruct-GPTQ-Int4 with four AD102s, ZhiLight's TTFT mean of 1111.8 is roughly a third of vLLM's 3493.97. Then the A800 four-GPU row for the same model goes the other way: SGLang posts 0.36 QPS and ZhiLight 0.18, with SGLang also ahead on TTFT and TPOT. Qwen1.5-110B-Chat-GPTQ-Int4 on four A800s is similar, with SGLang at 0.18 QPS and ZhiLight at 0.1. So the claim of "significant performance advantages" holds in the configurations the project chose to highlight and does not hold in others it also published. The README does not state batch sizes, sequence lengths, concurrency levels or client counts for these runs, and without those numbers the tables are directional rather than reproducible. Treat them as a reason to run your own test, not as a procurement document.

Where the design stops short

The README says tensor parallelism and pipeline parallelism are supported "on one node," with TP recommended. That is a hard ceiling. A model that does not fit across the GPUs in a single host needs a different engine, or a different deployment topology layered on top. The dual-stream and host all-reduce work is also specifically tuned for PCIe, so on an NVLink-connected node the advantage that motivates the whole project narrows. Quantization support is a list of formats, not a guarantee per model: the README does not say which combinations of architecture and quantization method have been validated, and a Marlin kernel for GPTQ does not imply Marlin coverage for AWQ or FP8. The OpenAI-compatible interface is described as "adapted from vllm," which tells you the API surface resembles vLLM's but says nothing about which endpoints, sampling parameters or streaming behaviours are implemented. There is no published statement about multi-tenant scheduling fairness, request cancellation, or what happens under memory pressure when the prefix cache fills. Finally, the repository has no homepage listed, and the roadmap lives in a GitHub wiki rather than in the tree, so planning information is thinner than the code surface suggests.

ZhiLight against vLLM and SGLang

The README names vLLM and SGLang as the comparison points, and the difference in approach is real rather than cosmetic. vLLM's core contribution is PagedAttention and a block manager that lets many concurrent sequences share KV cache memory without fragmentation; its scheduling and memory management are the product. SGLang builds on a similar base and adds a frontend language and runtime for structured generation and prefix reuse. ZhiLight takes a different route: it assumes the interconnect is the problem and spends its complexity on overlapping communication with compute, quantizing the all-reduce payload, and moving the reduction to the host when that is faster. Its memory story is a custom tensor type with unified global memory management rather than a paged block allocator, and its API compatibility is a surface adapted from vLLM rather than the vLLM codebase itself. The practical consequence is that you get less ecosystem, fewer integrations and a narrower hardware target, in exchange for a stack that has been tuned for one specific bottleneck. If your bottleneck is scheduling many short requests, vLLM's block manager is the more direct answer. If it is PCIe all-reduce during decode, ZhiLight is aimed at exactly that.

Maintenance, licensing and what to check before you commit

The project is Apache-2.0, which permits commercial use and modification and requires that you retain the licence and notices; it also includes an explicit patent grant. That is the permissive end of the spectrum and imposes no copyleft obligation on your own code. This is not legal advice, and if you redistribute a modified engine you should read the licence text and your own counsel's reading of it. On maintenance: the last push recorded for the repository is March 2026, and the most recent release listed is v0.4.8 from December 2024, so there is a gap between release cadence and repository activity that you should understand before pinning a version. The CUDA 12.4 image tag ties you to that toolkit generation unless you build from source, and building from source means a C++ toolchain, NCCL and cuBLAS on the build host. Upgrade cost is dominated by kernel and quantization compatibility rather than API churn, since the serving interface is meant to track vLLM's. The concrete first step is to pull ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124, run python -m zhilight.server.openai.entrypoints.api_server against one checkpoint from the supported list, and measure TTFT and TPOT on your own hardware at your own concurrency before reading anything into the published tables.

Editorial conclusion

Adopt ZhiLight if you serve Llama-family, Qwen or DeepSeek checkpoints on PCIe A800, H20 or AD102 nodes and you want host-side all-reduce and dual-stream overlap rather than a general-purpose scheduler. Skip it if you need a large plugin ecosystem, non-NVIDIA accelerators, or multi-node tensor parallelism, since the README states TP and PP are supported on one node only. Before committing, reproduce the v0.4.8-cu124 image against your own checkpoint and confirm the Qwen1.5-110B-A800 case, where the published table shows ZhiLight behind SGLang on QPS, is not the configuration you plan to run.

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. zhihu/ZhiLight on GitHub