mlxcel: a Rust runtime that loads MLX checkpoints without a conversion step
High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon and NVIDIA CUDA systems.
At a glance
- What is it?
- mlxcel is a Rust CLI and server from lablup that executes MLX SafeTensors checkpoints through native MLX C++ bindings on Apple Silicon, with Linux/CUDA as a secondary target. Its selling point is the absence of Python in the request path, and its main cost is that the project is still moving fast enough that main and the latest tag differ.
- Who is it for?
- Adopt mlxcel if you are serving MLX-format checkpoints on Apple Silicon and want a single native process instead of a Python interpreter plus a conversion pipeline, and if you are willing to pin a tag rather than track main. Do not adopt it if you need a stable long-term API surface, if your checkpoints are GGUF, or if your audio workloads use MP3 or FLAC on a family other than Phi-4 Multimodal or Gemma 3n.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The conversion step mlxcel removes
Most local inference stacks on Apple Silicon end up with two artifacts: the checkpoint you downloaded and the converted form the runtime actually reads. mlxcel collapses that. The README states that the CLI and server execute MLX SafeTensors checkpoints through native MLX C++ bindings, without Python in the request path or a checkpoint-conversion step. Checkpoints from HuggingFace, including the mlx-community organisation, load directly, and the README adds that many standard SafeTensors embedding checkpoints also load without conversion. That second claim is narrower than the first, and the wording is deliberate.
The audience follows from that. If you are packaging a model server into a container or a supervised service, a Python interpreter layer is something you have to provision, pin and patch. mlxcel ships two native executables, mlxcel and mlxcel-server, which the README describes as suitable for packaging and service supervision, with the caveat that platform runtime libraries are still required. The project is for engineers who already live in the MLX checkpoint ecosystem and want the serving layer to be a binary rather than an environment.
One process for scheduling, batching and caching
Loading, scheduling and inference stay in one process. Continuous batching, prompt-prefix caching and automatic prefix caching are on by default, so a deployment that does nothing beyond starting the server already gets those behaviours. Speculative decoding, KV-cache compression, router mode, live LoRA and distributed modes are opt-in and gated on model and backend support.
Router mode is the piece with real operational shape. It discovers checkpoints from a model store, a model directory, or INI presets, loads them on demand, and bounds the resident set with LRU eviction. That is a capacity decision expressed as configuration rather than as a process supervisor. LoRA adapters can stay unfused so per-request selection and live scale changes are possible, or be fused for zero decode overhead. The trade-off is explicit in the README: unfused adapters buy flexibility, fused adapters buy decode speed.
Speculative decoding is handled with more caution than the feature list suggests. The MTP paths for Gemma 4, Qwen and GLM-4.7-Flash, the LFM2 and LFM2.5 DSpark path, and the Muse Glimmer assistant path all probe greedy exactness before enabling the fast path, and the server exposes the adaptive decision at GET /v1/internal/mtp-policy. LFM2.5 targets pair with LiquidAI's published DSpark drafters via --draft-model and are greedy-decoding only. Muse Glimmer pairs with meta-models/Muse-Glimmer-30B-assistant on text-only requests, and the README notes the round loop measures its verify width rather than fixing it. mlxcel split-mtp extracts the GLM-4.7-Flash drafter from the raw checkpoint.
Install and first run
Homebrew is the shortest path on macOS and Linux:
brew tap lablup/tap brew install mlxcel
That installs the latest released mlxcel and mlxcel-server binaries. If you need unreleased work, the README directs you to install from source and to read CHANGELOG.md for the release-by-release record.
Running a model is a single command. mlxcel run resolves the model, downloads it on first use, reuses it afterward, and starts an interactive chat REPL. A bare model name resolves under mlx-community by default, so this works without a full repository path:
mlxcel run Qwen3.5-0.8B-4bit
The README also shows a one-shot generation form alongside the interactive one. Two other commands appear in the material and are worth knowing before you commit: mlxcel arch prints the binary's architecture catalog, which is the authoritative list for the build in front of you rather than for the documentation, and mlxcel split-mtp extracts the GLM-4.7-Flash drafter from a raw checkpoint.
For load-time weight edits, default builds support YAML surgery through --surgery or the MLXCEL_SURGERY environment variable, with the operations scale, add, prune, replace and interpolate. That is a reproducibility mechanism: the edit is a file you can version, not a script someone ran once.
Serving routes and the b10621 boundary
mlxcel-server accepts a broad llama-server flag surface and the LLAMA_ARG_* environment variables. The claim worth examining is the one about the checked-in manifest. The README states that a frozen b10621 manifest classifies all 376 pinned options, routes and native request fields, with no deferred entries, and that native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router and LoRA surfaces are implemented or explicitly classified rather than silently ignored.
Explicit classification is the interesting part. A server that accepts a flag and ignores it is worse than one that rejects it, because the failure is silent. The README points to docs/llama-server-compat.md for the exact boundary, and that file is where you should look before assuming a flag you rely on does what it does upstream. The word used is compatibility without overclaiming, and the manifest is the mechanism that makes that checkable.
Alongside the native routes, mlxcel-server exposes OpenAI, Anthropic and Vertex-compatible surfaces: Chat Completions, Completions, Responses, Embeddings, Reranking, Audio and Anthropic Messages. Optional Vertex AI custom-container routing goes through the standard AIP_* variables. Operational controls include optional authenticated live settings, prompt and cache and slot observability, bounded response stores, idle model sleep and wake, GBNF grammars, expanded sampling controls, runtime reasoning placement, and API-key, CORS and TLS controls.
Where the coverage is narrower than the feature list
Audio is the clearest example of a boundary stated in the README rather than discovered later. The compatible transcription boundary recognises WAV, MP3 and FLAC by content, and chat-model transcription streams one ASR delta per decoded token. But container support is model-dependent: Phi-4 Multimodal and Gemma 3n decode all three containers, while other current audio families and dedicated Whisper remain WAV-only. Kokoro provides text-to-speech. If your pipeline feeds MP3 to Whisper, this is the wrong tool for that step.
The backend story has a similar shape. Apple Silicon is the primary target, Linux/CUDA is secondary, and the opt-in OpenXLA/IREE backend is described as an alpha development path. Alpha is the project's own word. Treating it as a production backend is a decision the README does not support.
The third boundary is versioning. The current main branch is v0.7.0 plus unreleased work, and the README tells you to install from source for features that have not reached the latest tagged release. The release list shows v0.7.0 on 2026-09-09 and v0.6.0 on 2026-08-22, with a beta in between. A project shipping releases roughly every two to three weeks has a moving surface, and the gap between main and the tag is where surprises live.
There is also a precision detail worth reading closely. The README says activation helpers, the multimodal rotary, the Gemma3n load policy and the GraniteMoeHybrid gated norm no longer widen a half-precision hidden state to f32, which had made every later matmul promote its own weight. It reports the affected checkpoints move by up to 3.26x, and that the bridge's reductions accumulate in f32 to close a bfloat16 max that returned NaN for a finite input. That is a correctness fix as much as a speed one, and it is the kind of bug that only shows up on specific checkpoints.
Alternatives and the difference in approach
The obvious comparison is llama.cpp's llama-server, and mlxcel's own documentation invites it by pinning compatibility to a b10621 manifest. The difference is the checkpoint format and the execution path. llama-server runs GGUF, which requires converting from the original weights and quantising into that container. mlxcel runs MLX SafeTensors directly through MLX C++ bindings, and its README frames the design around having no checkpoint-conversion step and no Python in the request path. If your weights are already in MLX format from mlx-community, mlxcel skips a stage that llama.cpp requires. If your weights are GGUF, or you depend on the wider llama.cpp ecosystem of quantisation tooling, the conversion step is not overhead you can avoid by switching.
The second alternative is running MLX from Python. That gives you the full MLX Python API and the ability to write custom model code, at the cost of provisioning an interpreter and its dependencies in production. mlxcel's answer is a fixed set of supported architectures plus load-time surgery through --surgery and MLXCEL_SURGERY. If your model needs code that does not exist as a YAML edit, the native runtime is the constraint, not the solution.
Both comparisons come down to the same question: whether the checkpoint format you already have matches the runtime you want to operate.
Licence, maintenance and what to verify
mlxcel is Apache-2.0, which permits commercial use, modification and redistribution under the terms in the LICENSE file, including its patent grant and notice requirements. This is a description of the licence identifier, not legal advice; if you are redistributing modified binaries or embedding the runtime in a product, read the licence text and your own obligations.
The maintenance cost is dominated by release cadence. With v0.6.0 in late August and v0.7.0 in early September, the project is shipping meaningful changes on a short cycle, and the README explicitly separates tagged releases from main. Pinning a tag is the low-effort option. Tracking main means reading CHANGELOG.md and the compatibility document on each move, because the b10621 manifest is frozen against a specific llama-server revision and a flag's classification can change between releases.
Three things to verify before adopting. First, run mlxcel arch on the binary you actually intend to deploy and confirm your architecture is in that build's catalog, since the documentation covers the project rather than your specific tag. Second, read docs/llama-server-compat.md and find the options you depend on, checking whether each is supported, aliased, not applicable, or an intentional difference. Third, confirm whether the feature you need is in the tagged release or only on main, and if it is on main, decide whether building from source is something your deployment process can absorb. The model surgery operations and the MTP policy endpoint at GET /v1/internal/mtp-policy are the two places where behaviour is inspectable at runtime rather than only documented.
Editorial conclusion
Adopt mlxcel if you are serving MLX-format checkpoints on Apple Silicon and want a single native process instead of a Python interpreter plus a conversion pipeline, and if you are willing to pin a tag rather than track main. Do not adopt it if you need a stable long-term API surface, if your checkpoints are GGUF, or if your audio workloads use MP3 or FLAC on a family other than Phi-4 Multimodal or Gemma 3n. Before committing, run mlxcel arch on the exact build you intend to ship, read docs/llama-server-compat.md to see how the b10621 options you depend on are classified, and check whether the feature you need is in the tagged release or only on main.
Community notes