vllm-mlx: an OpenAI and Anthropic compatible inference server for Apple Silicon
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
At a glance
- What is it?
- vllm-mlx packages MLX inference behind OpenAI and Anthropic compatible endpoints, with continuous batching, paged KV cache and multimodal support. It is a good fit for Mac-based local serving and a poor fit for anyone without Apple Silicon.
- Who is it for?
- Adopt vllm-mlx if you are serving models on an M-series Mac and want your existing OpenAI or Anthropic client code to work without changes; the dual API surface and continuous batching are the reasons to pick it over calling mlx-lm directly. Do not adopt it on Linux, on NVIDIA hardware, or anywhere you need multi-node scaling, because the README states Apple Silicon only.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 10 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap vllm-mlx fills between mlx-lm and a real serving stack
Running a model on a Mac is easy. Serving one to several clients at once is not. The mlx-lm library gives you a generation loop, not a scheduler, so concurrent requests either queue or thrash memory. The README positions vllm-mlx against exactly that: "Unlike Ollama or mlx-lm used directly, it ships continuous batching, paged KV cache, prefix caching, and SSD-tiered cache." The target user is an engineer on an M-series Mac who wants to point an existing client at localhost and have it behave like a hosted endpoint. The clearest signal is the Claude Code recipe, which sets ANTHROPIC_BASE_URL to the local server and runs the claude CLI against it. That is a specific workflow: agentic coding against a local model, with tool calling, on hardware you already own. If you only ever run one prompt at a time from a script, the batching machinery is dead weight and mlx-lm is simpler.
Two API surfaces, one process, and what that costs
The server exposes OpenAI routes (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/responses) and an Anthropic route (/v1/messages) with streaming, tool use and system prompts. Both come from a single process, which is the main architectural claim. The practical consequence is that the same loaded weights serve a Python script using the OpenAI SDK and a Claude Code session using the Anthropic wire format, without a translation proxy in between. Tool calling is handled by what the README calls 19 parsers covering OpenAI, Anthropic, Gemini, Qwen, DeepSeek and Gemma formats. That parser count is worth reading carefully: model families emit tool calls in different textual shapes, and a server that only understands one shape breaks agent loops on other models. Structured output goes through lm-format-enforcer for JSON Schema via response_format. The trade-off is surface area. Supporting two API dialects plus nineteen tool parsers plus vision, audio, embeddings and reranking in one Python process means more code paths that can diverge, and the README does not describe a conformance test suite for either API.
Getting a server running and pointing a client at it
Installation is a single pip package, and the README's quick start is two lines: pip install vllm-mlx, then vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching. The OpenAI SDK connects by setting base_url to http://localhost:8000/v1 with any api_key string, and the model field is passed as "default". For the Anthropic side, export ANTHROPIC_BASE_URL=http://localhost:8000 and ANTHROPIC_API_KEY=not-needed, then run claude. Optional flags change what the server does: --reasoning-parser qwen3 extracts thinking into a separate reasoning field on the response message, --embedding-model loads a second model for /v1/embeddings, --metrics turns on a Prometheus /metrics endpoint, --ssd-cache-dir spills the prefix cache to disk for long-context agents, and --warm-prompts preloads popular prefixes at startup. The README claims 1.3 to 2.25x TTFT improvement from warm prompts but does not state the measurement conditions, so treat that range as unverified until you reproduce it. Audio needs an extra: pip install vllm-mlx[audio] plus brew install espeak-ng for non-English TTS.
Where the design assumptions break down
The hard constraint is in the feature list: Apple Silicon only, M1 through M5, with Metal kernels via MLX. There is no CUDA path and no CPU fallback described. That rules out the deployment most teams actually run, which is a Linux box with a discrete GPU. It also means capacity is bounded by one machine's unified memory. A 128 GB M4 Max is the configuration used for the published decode numbers, and the largest model in that table, Qwen3-30B-A3B-4bit, is listed at roughly 18 GB. You cannot scale out by adding a second node, because nothing in the README describes distributed serving. The reranker section is unusually candid about another boundary: the built-in MLX reranker forward path handles BERT and XLM-RoBERTa sequence-classification weights with gelu, gelu_new, gelu_fast, relu, silu or swish activations, and other activations "fail explicitly so custom reranker architectures can add a dedicated adapter instead of silently using the wrong activation." That is the right failure mode, but it does mean arbitrary reranker checkpoints will not load. The published throughput figures are also single-stream greedy decode on one specific machine, so they say nothing about behavior under the concurrent load that continuous batching exists to serve.
How it differs from Ollama and from running mlx-lm yourself
Ollama is the obvious comparison for local Mac inference, and the difference is scope. Ollama is a model runner with a management CLI and its own API; vllm-mlx is a serving layer that speaks OpenAI and Anthropic protocols natively and adds paged KV cache, prefix caching and an SSD cache tier. If your code already targets the OpenAI SDK, the base_url swap is the whole integration. If you are running mlx-lm directly, you own the request queue, the cache and the concurrency behavior; vllm-mlx takes those over and exposes them as flags. The cost of that convenience is a much larger dependency surface, including lm-format-enforcer, optional audio packages and espeak-ng. On the other side, the SSD-tiered cache is a feature neither Ollama nor a hand-rolled mlx-lm script offers, and it targets a real problem: agent sessions with long, repeated prefixes that would otherwise be recomputed or held in memory.
Version cadence, licence and what maintenance looks like
The repository is active, not archived. Recent releases are v0.4.1 in August 2026, v0.4.0 in June 2026 and v0.4.0rc1 in May 2026, with the last push to main in September 2026. That is a roughly quarterly minor cadence with release candidates before the stable tag, which suggests the maintainer treats API changes as breaking-worthy and stages them. The project is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant; it does not impose copyleft on your own code. That is a statement about the licence text, not legal advice, and if you redistribute the server or bundle it into a product you should read the NOTICE and attribution requirements yourself. The upgrade cost that matters here is model compatibility rather than API churn: a new MLX release or a new model architecture can require a new parser or forward path, which is why the reranker activation list is an explicit allowlist rather than a fallback.
Who should run it and what to check before you do
The fit is narrow and clear. You have an M-series Mac with enough unified memory for the model you want, you need more than one client hitting it at a time, and your existing code targets the OpenAI or Anthropic API. The multimodal and audio paths widen that: vision models including Gemma 3, Qwen3-VL, Pixtral and Llama vision, audio input as audio_url content blocks, TTS with 11 voices across 15 or more languages, and Whisper STT with RTF figures the README lists as up to 197x for whisper-tiny on an M4 Max. If you need any of that behind one endpoint, the alternative is stitching together several tools. What to verify first is model coverage, because the supported lists are explicit and a model outside them is a dead end. Then run the built-in benchmarker against your own traffic: vllm-mlx bench-serve --url http://localhost:8000 --workload workload.json --repetitions 5 --output results.json, using a workload file that resembles your real prompt lengths. The published tables are single-stream greedy numbers on one machine; your concurrency and context length are what determine whether continuous batching actually pays off for you.
Editorial conclusion
Adopt vllm-mlx if you are serving models on an M-series Mac and want your existing OpenAI or Anthropic client code to work without changes; the dual API surface and continuous batching are the reasons to pick it over calling mlx-lm directly. Do not adopt it on Linux, on NVIDIA hardware, or anywhere you need multi-node scaling, because the README states Apple Silicon only. Before committing, verify that your target model appears in the supported vision, audio or reranker lists, and run vllm-mlx bench-serve against your own prompt distribution rather than trusting the published M4 Max tables.
Community notes