ZhiLight: Zhihu's PCIe inference engine, its build typo, and the benchmarks it chose to publish
A highly optimized LLM inference acceleration engine for Llama and its variants.
At a glance
- What is it?
- A C++ inference engine from Zhihu and ModelBest aimed squarely at PCIe cards rather than datacenter interconnect, with an OpenAI compatible server, TP and PP on one node, and dual-stream encode and all-reduce overlap. The interesting parts are the build flags as actually documented, the way the published tables split by TTFT rather than throughput, and the gap between the release tag and the feature news.
- Who is it for?
- ZhiLight is worth evaluating on the one deployment shape it was written for: several PCIe consumer or data center cards on a single node, where the encode and all-reduce overlap and the fused decoding attention are designed to pay off. Its own tables show the gain concentrated in time to first token, and on datacenter parts at 72B and above the per-token latency can trail both vLLM and SGLang, so read the numbers for your card count rather than the summary line.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Activity is slowing. The repository last received commits 8 months ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- zhihu/ZhiLight906 ★ · -2
| Repository | 2026-09-12 | 2026-11-15 |
|---|---|---|
| zhihu/ZhiLight | 908 | 906 |
-2 Stars in the last 64 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The documented Ninja build sets an environment variable nothing reads
The build section is four lines long and one of them does not work. In setup.py the generator is read from a variable spelled with an O: `cmake_generator = os.environ.get("CMAKE_GENERATOR", "")`. The command shown for the Ninja backend spells the same variable without the O:
# Concurrently compile the wheel package, and turn off the unit test
CMAKE_BUILD_PARALLEL_LEVEL=32 TESTING=0 python setup.py bdist_wheel
# Compile with ninja backend
CMAKE_GENERATER="Ninja" python setup.py bdist_wheel
# Install directly
cd ./ZhiLight && pip install -e .
# Start OpenAI compatible server
python -m zhilight.server.openai.entrypoints.api_server [options]Because the lookup is by exact name, the Ninja line sets an environment variable that no code path reads. The build still succeeds, and it silently falls back to whatever generator CMake picks on its own, so the line looks like it did something and the speed difference people attribute to Ninja never appears. This is the kind of detail that costs an afternoon, and it is in the quickstart block rather than in a troubleshooting page.
The first line is the one worth copying. CMAKE_BUILD_PARALLEL_LEVEL=32 TESTING=0 turns off the unit test target and fans the compile across 32 jobs, which matters because the C++ side has to be built from source. That build step is the actual installation cost, and the README does not put it near the top of the document.
The same file also documents that setting Python_EXECUTABLE is required unless you use pybind11 with FindPython, which is the kind of note that saves a broken import on a machine with several Python environments.
The wins are in time to first token, not in tokens per second
The published comparisons have a shape that is easy to miss if you only read the bolded rows. Take Qwen2-72B-Instruct-GPTQ-Int4 on four AD102 PCIe cards:
| Engine | QPS | TTFT mean | TPOT mean | | --- | --- | --- | --- | | vLLM | 0.18 | 3493.97 | 35.47 | | SGLang | 0.18 | 2276.1 | 38.12 | | ZhiLight | 0.18 | 1111.8 | 26.75 |
Throughput is a three-way tie at 0.18 queries per second. What differs is the latency split: ZhiLight's mean time to first token is less than half of SGLang's and less than a third of vLLM's, and its time per output token is the lowest of the three. That is a coherent story for an engine that fuses encode work with the all-reduce and uses tensor core instructions for batched decoding attention. It is also a specific claim: faster prompt processing, not more concurrent requests.
The DeepSeek-R1 AWQ table on eight A800 cards shows the same asymmetry. vLLM sits at 0.08 QPS with a 109.86 mean per output token, while ZhiLight sits at 0.16 QPS with 115.97. ZhiLight roughly doubles throughput and takes longer per token, which fits a prefill-heavy workload.
So the honest summary is that ZhiLight trades steady-state decode speed for prompt handling and concurrency in most of its tables, and the design decisions named in the feature list, dual streams and flash attention prefill and chunked prefill, point the same direction. If your traffic is prompt-heavy and your answers are short, that is the right trade. If you are streaming long completions to many users at once, the per-token column matters more than the first-token column.
One benchmark table has no baseline at all
Not every table in the repository has a comparison row, and the one that does not is the DeepSeek-R1 AWQ table on eight H20 cards. It contains a single row:
| Engine | QPS | TTFT mean | TPOT mean | | --- | --- | --- | --- | | ZhiLight | 0.23 | 2025.15 | 128.28 |
There is no vLLM number and no SGLang number on that hardware, so the 0.23 QPS figure has nothing to be read against. The same model on eight A800 cards does have a vLLM row, and it shows ZhiLight at twice the throughput, which is encouraging but is a different card. H20 and A800 are both datacenter parts with different memory bandwidth characteristics, so carrying a result from one to the other is an assumption rather than a measurement.
This matters more than a missing row usually would, because the feature news leads with DeepSeek support. The news section announces FP8 block quantization for DeepSeek-V3 and R1 in May 2025, AWQ and GPTQ for the same models in February 2025, and DeepSeek-VL2 multimodal support in January 2025. DeepSeek support is the headline of the release history, and the headline hardware result is the one table without a baseline.
The project points to a benchmarks document under docs/benchmarks for more numbers, so the full comparison may exist outside the front page. On what the front page presents, the DeepSeek claim is supported by a single unaudited row.
The dependency list is heavier than the pitch for the Docker image
The container section makes a clean claim about what the engine needs. It says ZhiLight depends only on the CUDA runtime, cuBLAS, NCCL, and a few Python packages from requirements.txt, which is a reasonable pitch for an engine that otherwise would drag in a framework. Then there is the actual list:
torch>=2.4.1
numpy
transformers>=4.46.2
fastapi>=0.115.5
psutil>=6.1.0
prometheus_client>=0.21.0
uvicorn>=0.32.0
pybind11==2.13.6
flash-attn>=2.7.0.post2Two entries deserve attention. pybind11 is the only exact pin in the file, at 2.13.6, which is a reasonable thing to do for the binding layer but means an environment with a different version already installed will need a downgrade or a clean virtual environment. flash-attn is the other one: it is a compiled extension with its own CUDA build, and installing it is routinely a multi-minute operation rather than a wheel fetch.
The word few is doing some work in the Docker section. The list is nine entries, one exact pin, and one package that compiles against the local CUDA toolkit, so the practical dependency surface is closer to a research setup than to a pip install line. That does not contradict the CUDA-runtime claim, which is about native libraries rather than Python packages, but it does mean the container is the shortest route and the local build is the one that needs patience.
The entry point is FastAPI plus uvicorn plus prometheus_client, which tells you the server is a Python ASGI application wrapped around a C++ core, and that the metrics endpoint comes from the Python side rather than from a compiled server.
The build script's comments argue with themselves
setup.py carries a set of comments that read like a draft someone left in rather than documentation anyone would keep. The extension class opens with three lines explaining that it needs a sourcedir instead of a file list and that the name must be the single output extension from the CMake build, with a pointer to scikit-build for the multi-extension case. Then, inside the CMake argument list, three more:
- a note that the output directory is required for auto-detection and inclusion of auxiliary native libraries - a note that CMake lets you override the generator, followed by the observation that the code needs to check this, and that the value can be set with Conda-Build - a note to set Python_EXECUTABLE instead if pybind11 is used with FindPython from Python
The generator comment is the tell. The variable it describes is read three lines later, so the comment is stating a doubt about code that is visible in the same file. The MSVC aside on the build type line is phrased the same way, as a note that the value is not used there but carries no harm.
None of this affects behaviour. It does tell you something about how the file was produced, and for a repository whose selling point is a hand-tuned C++ core, the build glue reads as machine-assisted. That is worth knowing if you plan to maintain a fork: the comments are not a design record, and the actual build configuration lives in the CMakeLists.txt and the cmake directory at the repository root rather than in this script.
One release tag from 2024, a container on the same tag, and news through 2025
There is a single release in the history, v0.4.8, dated 10 December 2024. The news section runs to May 2025 and lists DeepSeek FP8 support in that month, DeepSeek AWQ and GPTQ in February 2025, DeepSeek-VL2 and Llama 3 support in January 2025. The last push to the branch is 18 March 2026.
The container section closes the loop on the version question. The one image tag offered is:
docker pull ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124So the only published image is the tag that matches the only release, which is the December 2024 build, pinned to CUDA 12.4. If the DeepSeek features from the first half of 2025 are not in that image, then pulling it gives you the 0.4.8 server rather than the engine the news describes, and the repository root does contain a docker directory with its own Dockerfile if you would rather build from the current branch.
There is a second consequence. With a single tag and no release cadence visible, there is no obvious rollback target and no changelog history tied to tags, even though the repository root carries a CHANGELOG.md. For a project whose advantage is measured in latency numbers, being unable to say which commit produced a given measurement is the more expensive gap. Anyone reproducing the tables above has to build from source at an unspecified commit.
Parallelism is single node only, and tensor parallelism is the recommended path
The parallelism story is narrow and stated plainly. Tensor parallelism and pipeline parallelism are both supported, but only on one node, and the feature list recommends tensor parallelism. That is a deliberate constraint rather than a missing feature, and it matches the PCIe framing: on PCIe cards the interconnect is slow, so the engine's fused kernels and the dual-stream overlap are doing work to hide communication cost, but a multi-node deployment would have to survive a much worse interconnect than the one it targets.
Dual streams is the named mechanism. Encode work and the all-reduce are overlapped, and the all-reduce itself is implemented on SIMD instructions, with an INT8 quantized all-reduce available to cut its cost further. For a tensor-parallel deployment the all-reduce is on the critical path of every token, so shrinking it is where the TTFT and TPOT gains in the tables come from.
The quantization coverage is broad and the model coverage is narrower than the marketing line suggests. Native INT8, SmoothQuant, FP8, AWQ and GPTQ are all supported, plus a Marlin kernel for GPTQ. Models listed as supported are Llama and Llama 2, Mixtral, the Qwen2 series and similar, with DeepSeek-V2 MoE and MLA and the DeepSeek-VL2 multimodal family announced in the news. Mixtral is an MoE model, DeepSeek-V2 MLA is not, and the changelog is where the current boundary sits.
The other serving features are the ones people expect from a modern engine and are worth confirming are present rather than assuming: dynamic batch, flash attention prefill, chunked prefill and prefix cache. Prefix caching is the one that changes how you think about repeated prompts, since shared system prompts stop being recomputed per request.
Editorial conclusion
ZhiLight is worth evaluating on the one deployment shape it was written for: several PCIe consumer or data center cards on a single node, where the encode and all-reduce overlap and the fused decoding attention are designed to pay off. Its own tables show the gain concentrated in time to first token, and on datacenter parts at 72B and above the per-token latency can trail both vLLM and SGLang, so read the numbers for your card count rather than the summary line. Before you build, check the environment variable name for your generator, install the pinned pybind11 version, and budget for compiling the flash attention wheel rather than assuming the Docker image is a shortcut. If you need a maintained release cadence or a documented rollback path, the single tag and the stale container tag are reasons to look elsewhere.
Frequently asked questions
What is the ZhiLight LLM inference engine?
It is an inference acceleration engine written in C++ by Zhihu and ModelBest, built for Llama and its variants and aimed at PCIe-based GPUs. It exposes an OpenAI compatible interface adapted from vLLM and supports quantization, prefix caching, chunked prefill and MoE.
How do I build ZhiLight from source?
Compile the wheel with CMAKE_BUILD_PARALLEL_LEVEL=32 TESTING=0 python setup.py bdist_wheel, then install with cd ./ZhiLight && pip install -e . and start the server with python -m zhilight.server.openai.entrypoints.api_server. The documented Ninja line sets CMAKE_GENERATER, while setup.py reads CMAKE_GENERATOR.
What CUDA version does the ZhiLight Docker image target?
The published image is ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124, so it is built for CUDA 12.4 and matches the repository's only release tag. Building from the docker/Dockerfile in the repository gives you the current branch instead.
Which quantization formats does ZhiLight support?
Native INT8, SmoothQuant, FP8, AWQ and GPTQ, with a Marlin kernel for GPTQ and an INT8 quantized all-reduce for tensor parallel deployments. DeepSeek-V3 and R1 block-quantized FP8 was added in May 2025, with AWQ and GPTQ support in February 2025.
Does ZhiLight support multi-node inference?
No. Tensor parallelism and pipeline parallelism are supported on one node only, and tensor parallelism is the recommended path. The repository also carries a Dockerfile you can refer to for building outside the published image.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/zhihu-zhilight)Star history chart for your README
[](https://hysenlabs.com/projects/zhihu-zhilight)Was this useful?