mlx-openai-server: an OpenAI-compatible API server for MLX models on Apple Silicon

The gap mlx-openai-server fills between MLX and OpenAI clients

MLX gives you model loading and inference on Apple Silicon. What it does not give you is an HTTP surface that an existing OpenAI SDK client can talk to without changes. mlx-openai-server is that surface. It is a FastAPI application that loads one or more MLX models and exposes them under the endpoint families OpenAI already defines: /v1/chat/completions, /v1/responses, /v1/images/generations, /v1/images/edits, /v1/embeddings and /v1/audio/transcriptions.

The intended reader is someone running local models on a Mac who wants to keep their client code. The README's own example constructs an OpenAI client with base_url set to http://localhost:8000/v1 and api_key set to not-needed. Any non-empty string works as the key. That is the whole integration story: change the base URL, keep the calls.

The project covers more than text. The supported model type table maps lm to mlx-lm, multimodal to mlx-vlm, image-generation and image-edit to mflux, embeddings to mlx-embeddings, and whisper to mlx-whisper. One process can therefore serve a coding model, a vision model and a transcription model, which is unusual for a local inference wrapper and is the main reason to look at this rather than a single-purpose script.

How the server routes a request to the right backend

The routing is driven by a model type declared at launch. Each launch mode pairs a --model-type value with a --model-path, and that pairing decides which backend library loads the weights and which endpoints accept requests. A launch with --model-type lm accepts chat and responses traffic. A launch with --model-type whisper accepts audio transcriptions. A launch with --model-type image-generation accepts image generation requests and takes an additional --config-name such as flux-dev plus a --quantize flag.

The API-facing name of a model is not fixed to the path. Requests can use the model path, the --served-model-name passed at launch, or a served_model_name field in a YAML config. That indirection matters when you swap a quantized checkpoint for a full-precision one and want the client configuration to stay put.

Sampling defaults are server-side and overridable per request. The option table lists --max-tokens at 100000, --temperature at 1.0, --top-p at 1.0, --top-k at 20 and --repetition-penalty at 1.0. Those are the values applied when a request omits the corresponding field, which means a client that sends nothing gets a very large default generation budget.

The repository layout points at queueing and concurrency machinery: the Makefile's run target passes --max-concurrency 1, --queue-timeout 300 and --queue-size 100. Those flags are not documented in the excerpted option table, so their exact semantics are something to read from the source before you tune them. What the Makefile does make clear is that concurrency is bounded by configuration rather than left unbounded, and that requests beyond the queue size are subject to a timeout.

Install and first request on macOS

The README states the requirements plainly: macOS on Apple Silicon and Python 3.11 or later. The pyproject file narrows the Python range to >=3.11,<3.13, so 3.12 is the upper bound in practice. The install sequence creates a 3.11 virtual environment and installs from PyPI with uv.

python3.11 -m venv .venv
source .venv/bin/activate

uv pip install mlx-openai-server

If you want the repository version rather than the released package, the README gives an install straight from GitHub, which is useful when a fix has landed on main but not in a release.

uv pip install git+https://github.com/cubist38/mlx-openai-server.git

Whisper transcription has one extra system dependency. The README says to install ffmpeg through Homebrew.

brew install ffmpeg

Now start a text model. The README's example uses a Qwen3 Coder checkpoint and passes two parser flags alongside the model path. Those flags are not optional decoration: the README states that without the reasoning and tool-call parsers, reasoning and tool-call output will not be parsed correctly.

mlx-openai-server launch \
  --model-type lm \
  --model-path mlx-community/Qwen3-Coder-Next-4bit \
  --reasoning-parser qwen3_moe \
  --tool-call-parser qwen3_coder

The server binds to 0.0.0.0 on port 8000 by default. Point a client at http://localhost:8000/v1 with any non-empty API key and issue a normal chat completion.

import openai

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="mlx-community/Qwen3-Coder-Next-4bit",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(response.choices[0].message.content)

The model field must match the model path, the served model name, or the YAML served_model_name. If it does not, the request will not resolve to a loaded model.

Where the Apple Silicon boundary actually bites

The hardest constraint is the platform. The README requires macOS on Apple Silicon, and the dependency list confirms why: mlx, mlx-lm, mlx-vlm, mlx-whisper, mlx-embeddings and mflux are all in the install set. There is no CUDA path, no Linux CPU fallback documented, and no container image mentioned in the README. If your deployment target is a Linux box with an NVIDIA card, this project is not a candidate regardless of how well the API compatibility fits.

The second constraint is memory. The README devotes a section to long context and Metal OOM, which tells you that pushing --context-length high on a large model is a known way to hit out-of-memory conditions on the GPU. The documentation addresses it rather than hiding it, but the practical ceiling is the unified memory on your machine divided across every model you load. A multi-model config serving a vision model and a text model simultaneously splits that budget.

The third is the single-process shape. Concurrency is bounded by flags like --max-concurrency, and the Makefile example sets it to 1. There is no replication or sharding story in the README. Scaling means running more processes on more Macs, and the server gives you no coordination for that.

One documentation gap worth naming: the README covers launch options, model types and endpoints, but it does not document rollback or downgrade steps between releases. If you pin a version and need to move back, you are reading release notes and the changelog rather than a documented procedure.

How it differs from mlx-lm's own server and from llama.cpp

mlx-lm ships its own server, and that is the closest comparison. The difference is scope. mlx-lm's server is built around language models. mlx-openai-server adds vision through mlx-vlm, image generation and editing through mflux, embeddings through mlx-embeddings, and transcription through mlx-whisper, all behind the same /v1 prefix and the same API key convention. If you only ever serve one text model, the extra surface is weight you do not need. The moment you want a Whisper endpoint next to a chat endpoint in one process, the single-purpose server stops fitting.

The other comparison people reach for is llama.cpp's server, which also exposes an OpenAI-compatible API. The difference in approach is the runtime: llama.cpp is built around GGUF weights and runs across CPU, CUDA and Metal, so it travels to Linux and NVIDIA hardware. mlx-openai-server is bound to MLX and therefore to Apple Silicon. That is a real trade: you give up portability and, in many cases, the broader quantization ecosystem, and you get a stack that is native to the Metal backend and covers more model categories than text.

The repository also lists mlx-omni-server among the phrases people search alongside it, which suggests the two get compared. Both target MLX behind OpenAI-style endpoints. The model type table here is the concrete thing to check when choosing: verify that the categories you need are in it before you commit to either.

Multi-model configs, licence terms and upgrade cost

The README documents a YAML config path for running multiple models, with served_model_name as a configurable field. The repository ships examples/config.yaml as a starting point. This is the configuration to use when you want one process to answer chat, embedding and image requests without restarting between them. The cost is memory: every loaded model occupies unified memory, and the README's Metal OOM section is the relevant reading before you add a third or fourth model to the file.

Licensing is straightforward. The package metadata declares MIT, and the repository carries a LICENSE file at the top level. MIT permits commercial use, modification and redistribution provided the copyright notice and permission notice are preserved. That is a statement about the licence text, not legal advice; if you redistribute the server inside a product, have counsel confirm the notice requirements and check the licences of the model weights you load, which are separate from the server's licence.

Upgrade cost is moderate. The pinned dependency ranges in pyproject are tight: mlx-lm is capped below 0.32, mlx-vlm below 0.5, mflux below 0.18, fastapi below 0.130. Tight caps mean an upgrade of the server is usually an upgrade of the whole MLX stack at once, and a mismatch between the server version and the MLX version is the likely failure mode. The release history shows v1.8.1, v1.8.0 and v1.7.1 across roughly a month in April and May 2026, and the last push to main was on 2026-09-14, so the codebase is still moving. Pin a version in your own environment rather than tracking main.

Editorial conclusion

Adopt it if you are on Apple Silicon and want OpenAI SDK clients to hit local MLX models without rewriting call sites, particularly for the multi-model YAML setup or Whisper transcription. Skip it if you deploy on Linux, on NVIDIA hardware, or need a cluster of replicas behind a load balancer, because the project targets macOS on Apple Silicon and a single process. Before committing, verify that your model type is in the supported table, check the context length you need against the Metal OOM guidance in the README, and confirm the parser flags for any reasoning or tool-calling model you plan to serve.

Frequently asked questions

Official sources

1. cubist38/mlx-openai-server on GitHub
2. Issues
3. License: MIT
4. README
5. Releases