vllm-mlx: the quick start installs 0.4.1 while the project sits at 0.5.0
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
At a glance
- What is it?
- A vLLM-style inference server for Apple Silicon that speaks both the OpenAI and Anthropic APIs from one process. Its walkthrough pins an older release than the current tag, keeps continuous batching behind a flag, and reports every speed number from one machine in greedy single stream.
- Who is it for?
- This is worth using if you want one local process that answers both the OpenAI and Anthropic client shapes on a Mac, and you are the kind of user who reads the flag list instead of assuming defaults. Verify three things before you commit to it.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 8 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- waybarrios/vllm-mlx1,655 ★ · +89
| Repository | 2026-09-10 | 2026-11-15 |
|---|---|---|
| waybarrios/vllm-mlx | 1.6k | 1.7k |
89 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The walkthrough pins a release two minor versions behind the current tag
The quick start is two lines, and the first one asks for a specific older release:
python -m pip install 'vllm-mlx==0.4.1'
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000The newest tag is v0.5.0 and pyproject.toml carries the same 0.5.0. The page does not hide the gap: it says the version in the walkthrough is the pinned reference release, not an automatically updated latest version. Stated that way it is honest, and it still means the two lines most readers copy exercise the older engine on a project that has shipped v0.4.0, v0.4.1 and then v0.5.0. New users are also pointed at a separate page for an isolated release install before this block is used, and that page assumes the environment is already activated when it runs.
Continuous batching heads the tagline and comes last in the walkthrough
The first item in the project's own summary is continuous batching, and it is the flag you are told to add after you already have a working first response. The instruction is to restart with `--continuous-batching` to try the feature and its cache options, and the next sentence confirms that the minimal command above uses the default engine without that flag. So a first run measures the unbatched path, and paged KV cache, prefix caching and SSD-tiered cache sit behind a linked guide rather than behind the default. The neighbouring switches behave the same way: `--ssd-cache-dir` to spill the prefix cache to disk for long-context agents, `--warm-prompts` to preload popular prefixes at startup, `--enable-metrics` for the /metrics endpoint, `--reasoning-parser` to separate reasoning from Qwen3 and DeepSeek-R1 output, plus `--moe-top-k`, `--enable-mtp` and `--specprefill` for the specialised paths.
The only dependency with a ceiling is mlx-lm, because 0.32 breaks the cache and batch APIs
The dependency list explains itself in comments, and one entry carries a hard stop. mlx-lm is the only dependency with an upper bound, written as `mlx-lm>=0.31.3,<0.32.0`, with the note that mlx-vlm needs 0.31.3 and that 0.32 breaks cache and batch APIs. mlx-vlm is held at >=0.6.5 because that version and later fix a Qwen3.5 sanitize double-run issue tracked as #674. transformers is held at >=5.0.0 because mlx-lm 0.30.5 and later require transformers 5.0, where a release candidate bug is fixed in the stable release. Everything else uses an open lower bound: mlx>=0.29.0, tokenizers>=0.19.0, huggingface-hub>=0.23.0, numpy>=1.24.0, pillow>=10.0.0, opencv-python>=4.8.0, torchvision>=0.23.0, psutil>=5.9.0, gradio>=4.0.0. The single refusal in the file is also the one that marks the cache layer as load-bearing.
Both API surfaces take a placeholder key, and only one example pins the loopback address
Neither protocol checks a credential. The Python client is built against a base URL of http://localhost:8000/v1 with api_key set to not-needed, and the Anthropic path is two exports and a CLI:
export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claudeThe quick start is the only example that constrains the interface with `--host 127.0.0.1`. The Anthropic example and the multimodal example pass `--port 8000` and leave the host out, and nothing in the text states what the default bind address is or that a key becomes necessary once the port answers from another machine. Both surfaces, `/v1/chat/completions` and `/v1/messages`, are open in the configuration these commands produce. The openai client is a separate matter: it has to be installed in your Python client environment, because the server package does not install it.
The coding CLI matrix is seven clients, one model and one afternoon
The compatibility section is the clearest statement of its own limits in the project. Seven clients are named as validated: OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI, and the embedded agent in OpenClaw. The sentence carrying the evidence says all seven completed a streamed tool interaction and an exact file edit in one local run with Qwen3.8-27B-4bit on September 19, 2026, then adds that results apply to the tested client versions and settings. That is one run, one day, one model, one interaction shape. A client that changes its streaming format or its file-edit protocol between versions sits outside that evidence, and the versions themselves live in a separate client acceptance matrix rather than on the main page.
Every speed number comes from one M4 Max in greedy single stream
The performance section names its conditions once, in the table caption: M4 Max, 128 GB, greedy, single stream. Within that, Qwen3-0.6B-8bit reaches 417.9 tok/s at 0.7 GB, Llama-3.2-3B-Instruct-4bit reaches 205.6 tok/s at 1.8 GB, and Qwen3-30B-A3B-4bit reaches 127.7 tok/s at about 18 GB. The speech-to-text table uses the same machine and reports real-time factor, from 197x for whisper-tiny through 55x for whisper-large-v3-turbo down to 24x for whisper-large-v3. Single stream is the case where batching matters least, and concurrency is the reason the server exists, so these figures describe its floor. The other claims follow the same pattern: warm prompts are given as 1.3 to 2.25x TTFT, and `--moe-top-k` as plus 7 to 16 percent on Qwen3-30B-A3B, each scoped to a model rather than to a workload.
The package calls itself alpha, and the tree carries the test and result files
The classifier line in pyproject.toml reads Development Status :: 3 - Alpha, on the same page as benchmark tables. The rest of that metadata sets Apache-2.0, requires-python >=3.10, Python 3.10 through 3.13, and Operating System MacOS, with acceleration restricted to M1, M2, M3, M4 and M5 through Metal kernels and unified memory. The repository itself is unusually open about its working files: pytest.ini, tests/, a pre-commit config, benchmarks/, and two committed result files at the root, gsm8k_qwen3_0.6b_results.json and vlm_benchmark_results.json. Four readmes sit side by side, README.md, README.es.md, README.fr.md and README.zh.md, and nothing in the tree says how they are kept in step. The last visible line of the main page stops partway through a sentence, at the words The built-in M, just after the /v1/rerank curl example.
The examples directory ships four assistant recordings in the four readme languages
The examples directory doubles as the evidence behind the multimodal claims. It contains four assistant recordings, assistant_bank_en.wav, assistant_callcenter_fr.wav, assistant_service_es.wav and assistant_support_zh.wav, which line up with the four languages the readmes are written in. Alongside them are scripts for the live microphone path (mic_live.py, mic_realtime.py, mic_transcribe.py), separate demos for image, text and video through both the OpenAI client and curl, mcp_chat.py and mcp_tool_use.py, and audio_separation_example.py. The feature claims are larger than the shipped assets: native text to speech with 11 voices across more than 15 languages from Kokoro, Chatterbox, VibeVoice and VoxCPM, and speech to text from the Whisper family. Those numbers describe models the server can load. What sits in the tree is four short wav files and a set of runnable scripts.
Editorial conclusion
This is worth using if you want one local process that answers both the OpenAI and Anthropic client shapes on a Mac, and you are the kind of user who reads the flag list instead of assuming defaults. Verify three things before you commit to it. Check whether continuous batching matters to your load, because the default engine does not use it and you have to restart with the flag. Check the mlx-lm ceiling, since the file refuses 0.32 for breaking cache and batch APIs, which is the layer everything else sits on. And check the acceptance matrix rather than the compatibility sentence, because seven clients were validated in one run with one model on one day. The alpha classifier in the package metadata is the honest summary of the maturity here, whatever the throughput table suggests.
Frequently asked questions
what is vllm mlx
vllm-mlx is a vLLM-style inference server for Apple Silicon Macs. Unlike Ollama or mlx-lm used directly, it ships continuous batching, a paged KV cache, prefix caching and an SSD-tiered cache, and exposes both the OpenAI /v1 routes and the Anthropic /v1/messages endpoint from a single process on Metal.
Which version of vllm-mlx does the quick start install?
The quick start installs vllm-mlx==0.4.1 and says so plainly: it is the walkthrough's pinned reference release, not an automatically updated latest version. The newest release is v0.5.0, and pyproject.toml also carries version 0.5.0.
Does vllm-mlx require an API key to serve models on a Mac?
No. The OpenAI example passes api_key set to not-needed, and the Anthropic path exports ANTHROPIC_API_KEY=not-needed. The quick start also pins the interface with --host 127.0.0.1, while the Anthropic and multimodal examples pass only --port 8000.
Which coding agents has vllm-mlx been validated against?
OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI and the embedded agent in OpenClaw. The stated evidence is one local run with Qwen3.8-27B-4bit on September 19, 2026, covering a streamed tool interaction and an exact file edit, and it applies to the tested client versions and settings.
What hardware and Python version does vllm-mlx target?
Apple Silicon only, named as M1, M2, M3, M4 and M5, using Metal kernels through MLX with unified memory and no model conversion step. The Operating System classifier is MacOS and requires-python is >=3.10, with classifiers through Python 3.13.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/waybarrios-vllm-mlx)Star history chart for your README
[](https://hysenlabs.com/projects/waybarrios-vllm-mlx)Was this useful?