vLLM-Kunlun: Running vLLM on Kunlun XPU Without Forking vLLM
vLLM Kunlun (vllm-kunlun) is a community-maintained hardware plugin designed to seamlessly run vLLM on the Kunlun XPU.
At a glance
- What is it?
- Baidu's community hardware plugin registers the Kunlun XPU as a vLLM platform through Python entry points, so the upstream server command stays intact. It is worth adopting only if you already own Kunlun3 P800 silicon and can pin the whole dependency stack.
- Who is it for?
- Adopt vLLM-Kunlun if you already run Kunlun3 P800 accelerators and want the standard vLLM OpenAI-compatible server rather than a vendor fork. Do not adopt it if your fleet is NVIDIA or AMD, or if you need the newest upstream vLLM features on demand, since the plugin tracks a specific vLLM version and the default branch is v0.25.1-dev.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What the plugin actually removes from your deployment
Running an open source inference engine on non-NVIDIA accelerators usually means maintaining a fork. You patch the attention backend, you patch the sampler, you patch the model loader, and every upstream release becomes a merge conflict. vLLM-Kunlun takes the other route: it is a Python package that registers itself as a vLLM platform plugin through entry points, following the RFC Hardware Pluggable referenced in the README. The README states plainly that there is "no need to modify vLLM source code."
The audience is narrow and specific. You need Kunlun3 P800 hardware and Ubuntu 20.04, per the prerequisites section. If that describes your rack, the plugin lets you keep using the vLLM command line and the OpenAI-compatible HTTP surface you already scripted against. If it does not, nothing here applies to you, and the project makes no attempt to pretend otherwise. This is a hardware enablement layer, not a general inference framework.
How the Kunlun backend attaches to vLLM
The mechanism is plugin registration rather than patching. Because the integration lives behind the platform plugin interface, vLLM's scheduler, block manager and API server remain the upstream ones, and the Kunlun-specific work sits in the backend that the plugin supplies.
The README lists several pieces of that backend. Piecewise Kunlun Graph provides graph optimization for inference. FlashMLA handles multi-head latent attention for DeepSeek MLA architectures. Tensor parallelism is supported for multi-device execution, and the quick start example exposes it as --tensor-parallel-size. Speculative decoding covers MTP for DeepSeek-V3.2 plus DFlash and EAGLE-style proposers.
Kernel work is visible in the changelog rather than the architecture description. The February 2026 notes mention fused MoE for small batches, optimized attention metadata building, and a claim that Multi-LoRA inference reaches 80 percent or more of non-LoRA performance. That last figure comes from the project's own release notes, not from an independent measurement, and the conditions behind it are not stated.
Installation is where the real constraints live
The prerequisites are the least negotiable part of this project. Hardware is Kunlun3 P800. The operating system is Ubuntu 20.04, which is old enough that you should check it against the rest of your software stack before planning anything.
PyTorch is not stock PyTorch. The README specifies PyTorch >= 2.5.1 as a KL3-customized xpytorch build, and points to the installation page for it. That means your torch wheel comes from a vendor channel, and the plugin's compatibility is tied to that build rather than to the version number alone. Python must be 3.10 or newer. The vLLM version must match what requirements.txt pins. Transformers is pinned at exactly 5.2.0, with a note that Qwen3.5 requires transformers 5.x.
Nothing in the supplied material shows a pip install line, a wheel index URL, or a Dockerfile. The installation steps live on the Read the Docs site, and you should read that page before assuming a clean pip install will work, because the xpytorch dependency almost certainly will not come from PyPI.
Serving a model: the documented command and its flags
The README gives a complete server invocation. It runs python -m vllm.entrypoints.openai.api_server with --host 0.0.0.0, --port 8356, --model pointing at your local model path, --gpu-memory-utilization 0.9, --trust-remote-code, --max-model-len 32768, --tensor-parallel-size 1, --dtype float16, --max_num_seqs 128, --max_num_batched_tokens 32768, --block-size 128, --distributed-executor-backend mp and --served-model-name.
Two flags are worth pausing on. Prefix caching and chunked prefill are both explicitly disabled with --no-enable-prefix-caching and --no-enable-chunked-prefill. Disabling chunked prefill caps how large a prompt you can admit in one scheduling step, and disabling prefix caching removes the reuse benefit for shared system prompts. Those are the documented defaults for this backend, and whether they are hard requirements or conservative starting points is not stated in the material provided.
The client side is unremarkable, which is the point. A curl request to /v1/chat/completions with model, messages and max_tokens works exactly as it would against any vLLM server.
Model coverage is broad but uneven across columns
The support tables are the most useful part of the README because they are specific. Generative coverage includes Qwen2, Qwen2.5, Qwen3, Qwen3-MoE, Qwen3-Next, Qwen3.5, Qwen3.5-Moe, Llama2, Llama3, Llama3.1, gpt-oss, Gemma4, GLM4.5, GLM4.5Air, GLM5, InternLM2, Seed-OSS, DeepSeek-R1, DeepSeek-V3, DeepSeek-V3.2 and Kimi-K2. Multimodal coverage lists Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3-VL-MoE, Gemma4, InternVL-2.5, InternVL-3.5 and InternS1.
Read the columns, not just the rows. LoRA is marked only for the Qwen2, Qwen2.5, Qwen3, Qwen3-Moe, Qwen3-Next, Llama2 and Llama3 rows. Quantization is blank for Gemma4, InternLM2 and Seed-OSS, and blank for every multimodal entry except the Qwen-VL family. gpt-oss has neither quantization nor Kunlun Graph marked. If your deployment depends on AWQ weights or an adapter, the table tells you immediately whether your combination is covered. The README also mentions 20+ models while the tables list more than that, so treat the tables as the authoritative list.
Where this is the wrong tool
The version coupling is the biggest limitation. The default branch is v0.25.1-dev, and the README directs you to requirements.txt for the matching vLLM version. A hardware plugin that patches or registers against a specific engine version cannot absorb upstream releases on your schedule. If your team needs a vLLM feature that landed after the version this plugin targets, you wait for the plugin.
The development branch is also ahead of the last stable release. Releases listed are v0.11.0 in March 2026, with v0.11.0rc1 and v0.10.1.1 in December 2025, while the default branch is v0.25.1-dev and the changelog describes v0.25.1 as under development. Version numbering between the release tags and the branch name does not line up in the material provided, so confirm which tag corresponds to which vLLM version before you plan an upgrade.
Hardware lock-in is absolute. There is no fallback path, no CPU mode, no partial support for other accelerators. And the documented environment is Ubuntu 20.04 with a vendor PyTorch build, which means your container base image and your driver stack are effectively chosen for you.
The alternative you are actually choosing between
The realistic comparison is not another Kunlun plugin. It is the fork-and-patch model that hardware vendors have traditionally shipped: a vendor-maintained copy of vLLM with the accelerator backend merged in. That approach can move faster on kernels and can ship features the upstream interface does not yet expose. It also means you track two version histories and rebase constantly.
vLLM-Kunlun bets on the opposite trade. The README frames it as the recommended approach for integrating the Kunlun backend within the vLLM community, adhering to the pluggable hardware RFC. The payoff is that your serving scripts, your Kubernetes manifests and your client code stay identical to a standard vLLM deployment. The cost is that you are constrained by what the plugin interface can express and by the vLLM version the plugin targets. For a team that already standardized on vLLM tooling, that is usually the better trade. For a team that needs a kernel the upstream interface cannot reach, a fork may still be the only option.
Maintenance cost, licensing and what to check first
The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant. This is not legal advice; read the LICENSE file and your own counsel's guidance, particularly if you redistribute the plugin alongside the vendor xpytorch build, which carries its own terms.
Upgrade cost is driven by the version matrix. Because the plugin must match a vLLM version and a transformers version (5.2.0 as stated), an upgrade is a coordinated change across vLLM, transformers, the plugin and the xpytorch build, not a single pip bump. The changelog shows active work on CI, including E2E tests, unit test CI and ruff format checks added in January 2026, which is the kind of infrastructure that makes those coordinated upgrades less painful over time.
Before you deploy, verify three things. First, that your Kunlun3 P800 driver matches the xpytorch build the installation page specifies. Second, that the vLLM version in requirements.txt is one you can accept for the lifetime of the deployment. Third, that your exact model and quantization format appear together in the support tables, since several rows support the model but not the quantization.
Editorial conclusion
Adopt vLLM-Kunlun if you already run Kunlun3 P800 accelerators and want the standard vLLM OpenAI-compatible server rather than a vendor fork. Do not adopt it if your fleet is NVIDIA or AMD, or if you need the newest upstream vLLM features on demand, since the plugin tracks a specific vLLM version and the default branch is v0.25.1-dev. Before committing, verify the exact xpytorch build for your KL3 driver, the vLLM version pinned in requirements.txt, and whether your target model appears in the support tables with the quantization format you intend to serve.
Community notes