vllm-ascend: Running vLLM on Ascend NPUs via a Hardware Plugin

The Problem: vLLM on Ascend NPUs

vLLM is designed around CUDA and other GPU backends. Ascend NPUs, Huawei's AI accelerators, have a different programming model and memory hierarchy. Running vLLM on Ascend used to require a separate fork or a manual port that would drift from upstream. vllm-ascend solves this by implementing the hardware pluggable interface proposed in vLLM RFC 11162. That interface decouples hardware-specific code from the core vLLM codebase. The plugin registers itself as a backend, so vLLM can dispatch to Ascend NPUs without changing the core engine. The intended user is an engineer or ML platform team that has Ascend hardware and wants to serve open-source models with vLLM's features, such as continuous batching and paged attention, without maintaining a private port.

How the Plugin Architecture Works

The repository is written primarily in C++, which suggests that the plugin provides custom operators and kernels that vLLM's Python layer calls into. The README describes it as adhering to the hardware-pluggable interface from the RFC. In practice, that means the plugin exposes a backend that vLLM discovers at runtime. The plugin handles device-specific operations like memory allocation, tensor transfers, and kernel launches on the Ascend NPU. This is different from a fork because the core vLLM code remains unchanged; the plugin is an optional dependency. The documentation mentions support for Transformer-like models, Mixture-of-Experts (MoE), Embedding, and Multi-modal LLMs. That breadth implies the plugin must implement not just basic attention but also MoE routing and multimodal encoders, which is a significant amount of custom code. The support matrix on the docs site is the authoritative source for which exact models and features work, and it is updated per release.

Installation and Getting Started

The README points to the official guide for each release. For example, the v0.23.0 release links to a specific documentation version. The installation process typically involves installing vllm and then installing vllm-ascend via pip, but the README does not give exact commands. The documentation at docs.vllm.ai/projects/ascend is the reference. The support matrix is a key resource; it lists which models and features are tested. The plugin version numbers align with vLLM versions, as seen in releases like v0.23.0, which corresponds to vLLM 0.23.0. This alignment is important because the plugin is built against a specific vLLM API. When you upgrade vLLM, you must also upgrade the plugin to a matching version. The release notes mention a first official version v0.7.3 in May 2025, and the project has moved quickly, with v0.23.0 by August 2026.

What the Support Matrix Tells You

The support matrix is not a marketing page; it is a practical constraint. It lists which models and features are verified to work on Ascend. For example, MoE and multi-modal support are mentioned, but not every model in those categories is guaranteed to work. The matrix is the first thing to check before adopting the plugin. If your model is not listed, you may need to test it yourself or contribute support. The README also mentions user stories from LLaMA-Factory, verl, TRL, and GPUStack, which show that the plugin is used in fine-tuning, evaluation, and reinforcement learning workflows. That suggests the plugin is not just for inference; it integrates with training frameworks that use vLLM as a serving backend.

The Cost of Being a Plugin

The main limitation is version lag. vllm-ascend releases trail vLLM core releases. For example, vLLM 0.23.0 was released earlier than the plugin's v0.23.0, which came in August 2026. If you need a new vLLM feature immediately, you may have to wait for the plugin to catch up. The plugin is community maintained, not an official Huawei product, though the README mentions collaboration with the vLLM community and a blog post about the hardware plugin practice. This means support is best-effort. The repository has a #SIG-Ascend Slack channel and a users forum, which are good resources, but they are not a commercial support contract. Another limitation is that the plugin is specific to Ascend; if you run a mixed cluster with GPUs and NPUs, you need to manage two different backends. The plugin does not abstract away the hardware differences; it only makes vLLM work on Ascend.

Comparing to Alternatives

The primary alternative is to use a forked version of vLLM that has Ascend support built in, which was the approach before the hardware plugin existed. Some vendors still maintain such forks. The difference is that a fork is a separate codebase that must be rebased on every upstream vLLM release. A plugin, by design, tracks vLLM releases more closely because it only contains the hardware-specific parts. Another alternative is to use a different inference engine that natively supports Ascend, such as MindIE or other Huawei-provided serving stacks. Those may have better performance on Ascend because they are optimized for the hardware, but they lack the ecosystem and features of vLLM. The trade-off is between using a general-purpose engine with a community plugin versus a hardware-specific engine that may be more optimized but less flexible.

Maintenance and Upgrade Path

The repository is active, with a release cadence of roughly every two to three months. The release notes show a pattern of release candidates followed by official releases. The project has a contributors page and a weekly meeting, which indicates an active community. The license is Apache-2.0, which is permissive for commercial use. However, the plugin depends on Ascend-specific software, such as the CANN toolkit and drivers, which have their own licenses. You need to account for that in your deployment. Upgrading the plugin is not a simple pip install; you must ensure that the plugin version matches your vLLM version. The documentation for each release includes an official guide, which likely includes upgrade steps. The support matrix is updated per release, so you should re-check it after upgrading to see if your models are still supported or if new ones have been added.

Editorial conclusion

Adopt vllm-ascend if you are an Ascend NPU user who wants to run vLLM without maintaining a custom fork. It is the community-recommended path, and the support matrix should be your first stop to check your models and features. Avoid it if you need the latest vLLM features immediately; the release cadence lags vLLM core, so verify that your required vLLM version is aligned with the plugin release. Before deployment, confirm your Ascend driver and CANN versions match the documented requirements, and test your exact workload, especially if you rely on multi-modal or MoE models.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vllm-project/vllm-ascend on GitHub