vLLM Metal: running vLLM on Apple Silicon through MLX

The gap vLLM Metal fills on a Mac

Upstream vLLM targets CUDA and, more recently, other server-class accelerators. Apple Silicon is not one of them. The README is direct about the split: upstream vLLM supplies the API server, the scheduler, and the paged block manager, while mlx_lm supplies the token-wise model layers. vllm-metal sits in between and owns what the README calls the request-aware attention path: the paged varlen kernel, M5 NAX prefill, and speculative decoding. That division matters because it tells you what you are and are not getting. You get the same request handling, continuous batching, and OpenAI-compatible serving surface as a normal vLLM deployment, because those components are literally upstream code. You do not get the CUDA kernels, and you should not expect them to be emulated. The audience is narrower than the phrase Apple Silicon suggests. It is people who already want vLLM's serving semantics on a Mac they own, typically one machine, typically a model that fits in unified memory. If your goal is simply to run a quantized model locally, this is more machinery than you need.

How the MLX and PyTorch lowering path is layered

The README states that the plugin unifies MLX and PyTorch under a single lowering path. Read that alongside the architecture paragraph and the picture sharpens: the layers come from mlx_lm, the orchestration comes from vLLM, and the attention implementation is the plugin's own code. The consequence is that a request does not traverse one framework. It enters through the vLLM API server, gets scheduled and assigned paged blocks by upstream code, is executed token-wise by MLX model layers, and hits a Metal attention kernel that vllm-metal maintains. The README's news entries show where the engineering effort has gone. The August 2026 entry says the plugin now uses M5 NAX tensor units to accelerate MHA, GQA, and MQA prefill. The April 2026 entry says the unified paged varlen Metal kernel became the default attention backend in v0.2.0. Both are attention-side changes, not model-side ones, which is consistent with the stated ownership boundary. The practical implication for a reader is that the attention backend is a moving part you may need to reason about: the README documents that it changed defaults once, and it does not document a rollback procedure for that change.

Installing it and what lands on disk

The README gives a single install command: curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash. According to the same section, this installs the vllm-metal plugin, vllm core, and related libraries under ~/.venv-vllm-metal, which the README calls the default location. Activating that environment with source ~/.venv-vllm-metal/bin/activate puts the vllm CLI on your path, and the README points to the official vLLM CLI guide for usage rather than documenting flags itself. Two constraints are stated explicitly and are worth reading twice. First, macOS 15 (Sequoia) or later on Apple Silicon. Second, native arm64 Python 3.12, with the README stating that Rosetta and x86_64 Python are not supported. That second point is the one that breaks setups in practice, because a shell that has been migrated from an Intel Mac can resolve python3 to an x86_64 binary without any obvious symptom until the plugin fails to load. The README does not document a manual or offline installation path, so the script is the only route it describes. It also does not document an uninstall step; the environment lives in a named directory, which at least makes removal a filesystem operation rather than a mystery.

The model matrix is the real constraint

The README says the project supports a growing set of models and defers the detail to docs/supported_models.md. That deferral is honest but it puts the burden on you. Nothing in the README describes a fallback for an unsupported architecture, and the architecture section implies why: layers come from mlx_lm, so a model that mlx_lm cannot express is not going to run here regardless of what vLLM's scheduler can do with it. The one concrete model example in the README is mlx-community/Qwen3.8-27B-8bit, described as a 27B hybrid SDPA plus GDN linear model served on a single Apple Silicon Mac. That is a useful data point about scale and about the kind of architecture the project has invested in, and it is also the limit of what the README tells you. It does not state memory requirements per model size, does not state a context length ceiling, and does not say what happens when a model exceeds available unified memory. Treat the supported model file as a prerequisite check, not a suggestion. If your model is not listed there, the correct assumption is that it does not work yet.

Where this is the wrong tool

The clearest failure mode is scale. Everything in the README describes one Mac. There is no mention of multi-node serving, tensor parallelism across machines, or any distributed configuration, and the hardware requirement is a single Apple Silicon machine running macOS. If your workload needs to grow by adding accelerators, this is not the path. A second boundary is the Python constraint: x86_64 Python is unsupported, full stop, and that rules out a range of container and CI images that ship an Intel interpreter. A third is version churn. The releases listed are v0.28.0 on 2026-09-01, v0.29.0 on 2026-09-11, and a dev build v0.29.0.dev20260914040516 on 2026-09-14, with the last push to the repository on 2026-09-14. That cadence is fast, and the README's own news entries show attention backends changing defaults between versions. Fast is not the same as unstable, but it does mean a pinned version is a reasonable default for anything you depend on. Finally, the README makes a performance claim for v0.2.0 (83x TTFT, 3.6x throughput versus v0.1.0) without stating the hardware, model, or measurement method. Treat it as a directional signal about kernel work, not as a number you can plan capacity around.

How it differs from llama.cpp and MLX's own server

The obvious alternative for local Apple Silicon inference is llama.cpp, which targets the same hardware with a different design. llama.cpp implements its own inference engine and its own GGUF model format, and its server is its own. vLLM Metal takes the opposite approach: it keeps vLLM's API server, scheduler, and paged block manager as upstream code and swaps in MLX for the layers. The practical difference is what you inherit. With vLLM Metal you inherit vLLM's request semantics, its paged attention memory model, and its CLI, which matters if you already run vLLM elsewhere and want the same client code against a Mac. With llama.cpp you inherit a narrower, more self-contained stack that tends to support a wider range of quantized models and does not require a specific Python version, because it is not built around one. MLX's own tooling is the third option and the closest in spirit, since mlx_lm is already the layer provider here. Choosing vLLM Metal over raw mlx_lm means you are specifically buying the vLLM serving layer, not the MLX compute. If you do not need continuous batching across concurrent requests or an OpenAI-compatible endpoint, that purchase is hard to justify.

Maintenance, licensing, and what to verify

The repository is not archived and the last push was on 2026-09-14, the same day as the dev release, so the project is being worked on now. The release tags line up with that: v0.28.0 and v0.29.0 are days apart, and dev builds are published between them. For upgrade cost, the README gives you one lever and no others. The install script writes into ~/.venv-vllm-metal, so upgrading means re-running it or managing that environment directly; the README does not document a version pinning mechanism, a rollback procedure, or a changelog beyond the release tags and the news list. If you need reproducibility, plan to snapshot the virtualenv yourself. On licensing, the repository is Apache-2.0, which is permissive and includes an explicit patent grant, but note that this covers vllm-metal only. The README states that the install script also pulls in vllm core and related libraries, and those carry their own licences, as do the model weights you load from mlx-community. Apache-2.0 on the plugin does not settle the terms of what it installs alongside. None of this is legal advice; read the licences of the full dependency set before shipping. The first thing to verify is that your target model appears in docs/supported_models.md.

Editorial conclusion

Adopt vLLM Metal if you are serving a supported model on a single Apple Silicon Mac and want the vLLM API surface, the scheduler, and the paged block manager rather than a bespoke server. Do not adopt it if you need multi-GPU scaling, CUDA, or a model outside docs/supported_models.md, and do not expect x86_64 Python to work under any configuration. Before committing, verify three things on your own machine: that your model appears in the supported matrix, that your Python is native arm64 3.12 on macOS 15 or later, and that the attention path you need is actually the default in the release you install, since the README dates the unified paged varlen kernel to v0.2.0 and the latest tagged release is v0.29.0.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vllm-project/vllm-metal on GitHub