MLX Omni Server: an OpenAI and Anthropic compatible server for Apple Silicon

The problem MLX Omni Server solves on a Mac

Apple Silicon Macs can run quantized language models locally through MLX, but the tooling around that is mostly Python scripts and notebooks. If you have an application written against the OpenAI Python SDK, pointing it at a local model normally means rewriting the client layer or standing up a translation shim yourself. MLX Omni Server is that shim, packaged as a server.

The README frames it as a drop-in replacement, and the mechanism behind that claim is narrow: the server exposes the same route shapes as the hosted APIs, so the SDK's own request and response parsing keeps working. The README's quick start shows a client constructed with base_url set to http://localhost:10240/v1 and an api_key value of not-needed. Nothing else in the calling code changes.

The intended audience is developers who work on M1 through M4 machines and want inference to stay on the laptop. The README states that all processing happens locally, which matters when prompts contain code, customer records, or anything you would rather not send to a third party. It is not a serving stack for a team of fifty. It is a single-machine server for development, local tooling, and privacy-sensitive prototypes.

How the dual API layer is put together

The project is a FastAPI application, started through the mlx-omni-server console script that pyproject.toml maps to mlx_omni_server.main:start. Underneath, the chat path delegates to mlx-lm, vision to mlx-vlm, speech to text to mlx-whisper, text to speech to mlx-audio and f5-tts-mlx, image generation to mflux, and embeddings to mlx-embeddings. Those are the actual dependencies listed in pyproject.toml, and they explain why the install is heavier than a chat-only server would be.

The two API families live on different prefixes. OpenAI-shaped routes sit under /v1, covering chat completions, speech, transcriptions, image generations, embeddings, and model listing. Anthropic-shaped routes sit under /anthropic/v1, covering messages and a paginated model listing. The README's endpoint tables mark all of these as supported, and both chat routes carry tools, streaming, and structured output. The Anthropic messages route additionally lists thinking mode.

Model handling is the other half of the design. The README describes auto-discovery of MLX models already present in the Hugging Face cache, on-demand loading with caching, and automatic downloading when a requested model is not present. That means the first request for a model can block while weights are fetched, and memory grows with each model kept resident. The README does not document an eviction policy or a cache size limit, so treat concurrent multi-model use as unverified.

Function calling is handled with model-specific parsers according to the README, which is an honest admission that tool-call output formats differ between model families. Structured output goes through JSON schema validation, and the dependency list pins outlines at exactly 1.0.4, so that path is tied to a single version of the constrained-decoding library.

Installing MLX Omni Server and making a first request

The README gives a single install command. It requires Python 3.11 or newer, and pyproject.toml declares requires-python >=3.11 with a classifier for 3.11 only.

pip install mlx-omni-server

Start the server with no arguments and it listens on port 10240, which the README calls the default.

mlx-omni-server

For a different port or more verbose logs, the README shows these two forms. The log level is read from the MLX_OMNI_LOG_LEVEL environment variable.

mlx-omni-server --port 8000
MLX_OMNI_LOG_LEVEL=debug mlx-omni-server

Now point an OpenAI client at it. The model string in the README example is a four-bit quantized Gemma 3 build hosted under mlx-community, and the server will download it on first use if it is not already in your Hugging Face cache.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:10240/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

The Anthropic path is the same idea with a different base_url and SDK. Note the /anthropic prefix and the max_tokens argument, which the Anthropic messages API requires.

import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:10240/anthropic",
    api_key="not-needed"
)

message = client.messages.create(
    model="mlx-community/gemma-3-1b-it-4bit-DWQ",
    max_tokens=1000,
    messages=[{"role": "user", "content": "Hello!"}]
)
print(message.content[0].text)

If the first request hangs, the README's troubleshooting section suggests pre-downloading the weights with huggingface-cli download and confirming the MLX import works with python -c "import mlx; print(mlx.__version__)".

Where MLX Omni Server stops being the right tool

The hard boundary is hardware. The MLX dependency is declared with a darwin platform marker in pyproject.toml, and the README lists an Apple Silicon Mac as a requirement. There is no documented Linux or Windows path, so this cannot back a shared staging environment running on x86 or on a cloud GPU box. If your deployment target is a container on a rented instance, stop reading here.

The second limitation is single-machine scope. Nothing in the README describes authentication, rate limiting, request queuing, or multi-user isolation. The api_key value in the examples is literally not-needed, which is the clearest signal that this is not hardened for exposure beyond localhost. Binding it to 0.0.0.0 as the development instructions do makes it reachable from your network, and the README does not discuss what that implies.

The third is memory. On-demand loading with caching means each distinct model you request stays in unified memory until something releases it. The README does not document an unload endpoint or a maximum resident model count, so a workflow that rotates between a chat model, a Whisper model, and an image model can push a 16 GB machine into swap. Test with the specific models you intend to use rather than assuming the small Gemma example generalizes.

Finally, the project self-describes as Beta in its classifier. The README does not document rollback behaviour for a failed model download, and there is no mention of a health endpoint or metrics. For a local development server that is acceptable; for anything with an uptime expectation it is not.

MLX Omni Server compared with Ollama

The obvious alternative for local OpenAI-compatible inference is Ollama, and the difference is not quality but where the abstraction sits. Ollama ships its own model registry and a Go runtime with its own quantization and loading code, and it runs on macOS, Linux, and Windows. Its OpenAI compatibility layer is an adapter over that runtime.

MLX Omni Server takes the opposite approach. It does not implement inference itself; it delegates to the MLX family of libraries and to Hugging Face for model distribution. That means the models you can run are the ones published in MLX format, typically under the mlx-community organization, rather than a curated registry. In exchange, you get the MLX kernels tuned for Apple's unified memory architecture and direct access to whatever the mlx-lm, mlx-vlm, and mlx-audio projects support.

The practical consequence: if you need the same server binary on a Linux CI box and a developer laptop, Ollama is the only one of the two that can do that. If you are Mac-only and specifically want MLX-format weights, MLX Omni Server is the more direct route, and it is the only one of the two that also exposes an Anthropic-shaped messages endpoint with thinking mode according to its README.

Maintenance, licence, and what upgrading costs

The repository is not archived. The last push was on 2026-05-09, which is the same date as the v0.5.3 release, so the release and the last commit line up. Before that, v0.5.2 landed on 2025-12-21 and v0.5.1 on 2025-10-14. That is a cadence of a few releases a year, not weekly churn, and the gap between the most recent release and today should be read as a signal about how fast fixes arrive.

The upgrade cost is dominated by the dependency pins. FastAPI is constrained to >=0.117,<0.140, uvicorn to >=0.34.0,<0.35, mlx and mlx-lm to >=0.31.2,<0.32, and outlines is pinned to exactly 1.0.4. If another package in your environment needs a different FastAPI minor, you will be resolving that conflict by hand. The MLX pin is the tighter constraint in practice, because MLX releases move quickly and a minor bump upstream can require a matching server release.

Licensing is MIT, and the README states the project is not affiliated with OpenAI, Anthropic, or Apple. MIT is permissive, so embedding the server in a commercial product is permitted under the licence terms as written. What the licence does not cover is the models you load: those carry their own licences on Hugging Face, and some are not permissive. Check the model card separately. This is a description of the licence text, not legal advice.

Editorial conclusion

Adopt MLX Omni Server if your team develops on M-series Macs and already has code written against the OpenAI or Anthropic SDK, since the base_url swap is the whole migration. Do not adopt it for Linux servers, CI runners, or anything that must run on non-Apple hardware: the MLX dependency is gated to darwin in pyproject.toml. Before committing, verify that the model you need is available in MLX format on Hugging Face, check that Python 3.11 or newer is installed, and confirm the audio and image endpoints pull the extra dependencies listed in pyproject.toml rather than only the chat stack.

Frequently asked questions

Official sources

1. License: MIT
2. madroidmaq/mlx-omni-server on GitHub
3. Project website
4. README
5. Releases