xLLM: A C++ Inference Engine Built Around Chinese AI Accelerators
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
At a glance
- What is it?
- xLLM is an Apache-2.0 inference engine from JD.com, now hosted by the OpenAtom Foundation, that targets Ascend NPU, Cambricon MLU, Moore Threads, Hygon DCU, MetaX and Iluvatar hardware. The README is explicit about that hardware focus and much less explicit about everything else, which is the main thing to weigh before adopting it.
- Who is it for?
- Adopt xLLM if your serving fleet is Ascend NPU, Cambricon MLU, MUSA, DCU, MACA or Iluvatar CoreX hardware and you want a C++ engine that already names those backends rather than one you would have to port yourself. Do not adopt it if you are on NVIDIA CUDA or AMD ROCm, because the README lists no such target, and do not treat the day-0 model announcements as a support guarantee.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem xLLM solves is accelerator diversity, not raw speed
Most open source inference servers assume one vendor's hardware. If your procurement decision was made by someone who bought Ascend NPUs, Cambricon MLUs or Hygon DCUs, the standard serving stack is not an option and you are left writing kernels. xLLM's README states the project is 'specifically optimized for Chinese AI accelerators', and the hardware table names six families: Ascend NPU (A2, A3), Cambricon MLU, Moore Threads MUSA (S5000), Hygon DCU (BW1000), MetaX MACA (MXC500) and Iluvatar CoreX (BI150). That table is the whole pitch. The engine covers LLM, VLM, DiT and REC model classes, and the repository topics name DeepSeek, GLM and Qwen as the model families in view. The intended user is an infrastructure team inside an organisation that already owns this hardware and needs a serving layer that does not start with a porting project. It is not aimed at a team renting NVIDIA instances, and nothing in the material suggests it wants to be.
Service and engine are separate processes, with a graph-based execution core
The README describes a 'Service-Engine Decoupled Architecture': the service layer handles scheduling and availability, the engine layer handles computation. That split is the main architectural claim, and it matters operationally because it means the request scheduler can be restarted, scaled or replaced without touching the process that holds model weights and KV state. Underneath, the acknowledgment section credits ScaleLLM for the graph construction method and says xLLM references its runtime execution, so the execution path is a captured graph rather than an eager interpreter loop. KV cache is a second borrowed component: the project says it built hybrid KV cache management on top of Mooncake, 'supporting global KV cache management with intelligent offloading and prefetching'. That is a prefix-cache design that spans instances rather than one that lives and dies inside a single process. HTTP serving comes from brpc, tokenization from tokenizers-cpp, weight loading from safetensors, and structured output parsing from a C++ JSON parser the project says was written with reference to Python and Go implementations. The dependency list is short and each entry maps to a named job, which is a reasonable sign that the codebase is not carrying a second framework inside it.
Getting it running means reading the docs site, not the README
This is the weakest part of the repository for an outside evaluator. The README has a Getting Started section that is five links and no commands: Quick Start, Launch xLLM, Online Service, Offline Inference and Supported Models, all pointing at docs.xllm-ai.com. There is no install line, no build flag, no config key and no example request in the material available here. What the README does give is a container image at quay.io/repository/jd_xllm/xllm-ai under the tags tab, which is the most concrete deployment path on offer. For specific models it links per-model scripts rather than a general recipe: run_glm_53_flash.sh for GLM-5.3-Flash, run_minimax_m3.sh for MiniMax-M3, and run_deepseek_v4.sh for DeepSeek-V4. Those live on preview branches, not on main. Anyone planning a pilot should budget time for the docs site and treat the container tags as the starting point, because the repository itself will not tell you how to launch a server.
Day-0 model announcements are a support claim, not a support guarantee
The news list is the most visible part of the README and the easiest to misread. It records day-0 support for GLM-5.3-Flash, MiniMax-M3, DeepSeek-V4, GLM-5, GLM-4.7, GLM-4.6V and the GLM-4.5/4.6 series, plus VLM-R1. Day-0 support in this context appears to mean a deployment script exists on a preview branch at the moment the model lands. It does not tell you whether the path is validated on all six accelerator families, whether it covers the full context length, or whether it survives the next upstream checkpoint revision. The GLM-5 entry points at a deployment document in the zai-org/GLM-5 repository rather than in xLLM's own tree, which means that particular recipe is maintained elsewhere. If a specific model is why you are evaluating xLLM, verify it against the Supported Models page and the branch script for your accelerator, not against the news entry.
The hardware table is also the exclusion list
Nowhere in the README is CUDA mentioned. There is no NVIDIA row, no ROCm row, no mention of an AMD or Intel target. That is not an oversight in a project whose stated purpose is Chinese accelerator support, but it does mean xLLM is the wrong tool for the majority of teams evaluating inference engines today. A second constraint is version sensitivity on the one backend that does carry a version note: Ascend support is listed as 'HDK Driver 25.2.0 +', so a fleet on an older driver is out until it upgrades. The other five rows have blank Remark cells, which leaves their driver and toolkit prerequisites unstated in the material. A third issue is the release cadence. v0.10.0 shipped 2026-07-01, v0.10.1 on 2026-07-14, and v0.9.1 before that on 2026-04-14. Two releases two weeks apart followed by a three-month gap is a pattern worth noting if you pin versions, because it suggests the patch release was a fix rather than a scheduled cut. There is also no upgrade or migration documentation in the material, so a version bump means reading the release notes and the diff yourself.
Against vLLM, the difference is the backend, not the feature list
The obvious comparison is vLLM. Both are inference servers with a scheduler, paged or hybrid KV cache, and a graph-based execution path. The difference is where the optimisation work went. vLLM targets NVIDIA GPUs first and treats other backends as ports; xLLM's README inverts that and lists six non-NVIDIA accelerator families as the primary targets, with the Ascend path carrying a named driver floor. If you are on CUDA, vLLM is the lower-risk choice and xLLM offers you nothing. If you are on Ascend or MLU, the comparison flips: vLLM's coverage of those parts is not something the xLLM material speaks to, and the practical question becomes whether you want to maintain a backend port or use an engine that already claims the target. The Mooncake-based global KV cache is the second axis. vLLM's prefix caching is primarily per-instance; xLLM's README describes offloading and prefetching across a global cache, which is a different design with different failure modes around cache coherence and network latency. That is a claim from the README, not something verified here.
Maintenance cost and the Apache-2.0 position
xLLM is Apache-2.0, the same licence as vLLM, which permits commercial use, modification and redistribution provided the licence and notices are preserved. It is not a copyleft licence, so linking it into a proprietary serving stack does not by itself trigger source disclosure. That is a general description of Apache-2.0 and not legal advice; the file headers in the repository carry the full text. The maintenance picture is more complicated than the licence. The project was donated to the OpenAtom Foundation in July 2026, which moves governance away from a single company, though the README still credits JD Co. in the copyright header and says the engine is 'battle-tested at scale across JD.com's core retail business'. That operational claim is the strongest evidence in the material and it is also unverifiable from outside. The dependency chain is a real cost: Mooncake, brpc, tokenizers-cpp, safetensors and concurrentqueue all have to build against your accelerator toolchain, and a break in any of them lands on you. The acknowledgment list also names six university labs as collaborators, which suggests the project expects academic contributions to carry part of the backend work rather than a single vendor funding all of it.
Editorial conclusion
Adopt xLLM if your serving fleet is Ascend NPU, Cambricon MLU, MUSA, DCU, MACA or Iluvatar CoreX hardware and you want a C++ engine that already names those backends rather than one you would have to port yourself. Do not adopt it if you are on NVIDIA CUDA or AMD ROCm, because the README lists no such target, and do not treat the day-0 model announcements as a support guarantee. Before committing, check the supported models page at docs.xllm-ai.com against the exact checkpoint you plan to serve, and confirm the HDK driver version your nodes run, since the README ties Ascend support to HDK Driver 25.2.0 and above.
Community notes