SwiftLM: an MLX inference server in Swift, and where its own benchmarks say it breaks

The problem SwiftLM is aimed at, and who actually has it

Running an MLX model on a Mac usually means Python. That brings the interpreter, the GIL, and a dependency tree that has to be resolved before the first token. SwiftLM's stated position is the opposite: a native Swift server compiled to a single binary, with no Python runtime and, in the README's phrasing, no unnecessary memory copies. The target user is someone who already has an Apple Silicon machine and wants to talk to a local model over HTTP using the OpenAI client they already have in their code.

The second audience is narrower. The repository describes SSD streaming for 100B+ MoE models, with a flag to stream expert layers directly from NVMe when the model does not fit in unified memory. That is a specific claim about a specific hardware class: Macs where the weights exceed RAM. The README frames it as for 24 GB Macs, which is where the trade-off becomes visible in the numbers rather than in the marketing.

What the server actually does between your request and a token

The architecture visible in the material is a Swift binary linking MLX plus a compiled Metal kernel library. The build script compiles mlx.metallib from Metal kernel sources, and the release tarball ships that file alongside the binary, which is why the archive is described as self-contained. Inference runs on the GPU through MLX; the Swift layer owns the HTTP surface and the sampling loop.

Two mechanisms sit on top of that base. MTP speculative decoding batches multiple tokens for verification in one forward pass, which the README explains amortizes KV reads across the batch. TurboQuant compresses the KV cache, and the memory tables show what that means concretely: at 40K context on the 4-bit model, OS RAM drops from 48.7 GB to 18.2 GB. SSD streaming is the third mechanism, and it is the one with the clearest cost, since expert layers are read from NVMe during generation rather than held in RAM.

The README is explicit that the model is a Mixture-of-Experts with roughly 4B active parameters per token out of 26B, and that vanilla means all experts loaded into unified RAM. That distinction drives every benchmark table: the same flag combination helps or hurts depending on whether the workload is compute-bound or bandwidth-bound.

Getting a server up: the two paths and the flags that matter

The fastest path is the prebuilt tarball from the Releases page. Unpack it and run the binary with a model and a port:

tar -xzf SwiftLM-<version>-macos-arm64.tar.gz ./SwiftLM --model mlx-community/Qwen2.5-3B-Instruct-4bit --port 5413

Building from source is a single script that initializes submodules, installs cmake via Homebrew if missing, compiles mlx.metallib, and builds the release binary:

git clone --recursive https://github.com/SharpAI/SwiftLM cd SwiftLM ./build.sh

After that, the documented invocation is .build/release/SwiftLM with --model and --port, and models download automatically if they are not cached. The interesting flag is --stream-experts, which the README says to add when running oversized MoE models to bypass macOS virtual memory swapping and stream expert layers from NVMe SSD.

Note the shape of the interface. Model selection, port, and streaming are command-line flags, not API parameters. Anything you want to tune per deployment lives in the launch command, which means a container or service definition has to carry those flags rather than a request body.

The benchmark tables contain two configurations that get slower, and that is the useful part

The published results are unusually candid. On the 4-bit MoE model, SSD Stream is listed at 10.8 tok/s at 512 context and 9.0 tok/s at 100K. Adding TurboQuant to SSD streaming makes it worse, not better: 2.5 tok/s at 40K and 1.6 tok/s at 100K. The README does not explain that regression, and it should, because the combination looks reasonable on paper. A reader who assumes compression plus streaming is strictly better will pick the slowest row in the table.

The second counterintuitive result is MTP on 4-bit. The README states plainly that the 4-bit model is compute-bound on MoE expert dispatch, so batch verification scales linearly with token count and MTP provides no net throughput gain. On the 8-bit model, which the README calls bandwidth-bound, MTP gives +20% at 40K and +51% at 100K. Same flag, opposite outcome, decided by quantization level.

TurboQuant is the one change that improves both axes in every table shown. At 100K on the 4-bit model it is listed at 66.9 tok/s against 27.5 tok/s vanilla, and GPU allocation at 40K drops from 54.8 GB to 23.9 GB. The README also notes that TQ plus MTP undercuts TQ alone, since removing the KV bandwidth bottleneck leaves the FFN compute untouched while MTP adds overhead. These are all vendor benchmarks on one machine, an M5 Pro with 64 GB, and the README does not describe the methodology beyond a profile_runner.py invocation and a run_benchmark.sh option number. Treat the ratios as directional.

What the OpenAI-compatible claim does not cover

The README calls the API strict and OpenAI-compatible, and stops there. There is no endpoint list, no schema, no statement about which OpenAI features are implemented and which are not. Streaming responses, tool calling, logprobs, and the various sampling parameters are all unspecified in the material. If your client depends on function calling or structured output, you cannot confirm support from what is published.

The same gap applies to the iOS side. The repository description mentions a macOS and iOS iPhone app, and the README embeds a demo GIF labeled SwiftBuddy, but there is no build instruction, no App Store link, and no statement about whether the iOS app is the same server or a client of it. The topics list includes on-device-ai and ios, which suggests the app runs inference locally, but that is inference from metadata rather than documentation.

A third limitation is hardware. Every benchmark is on one Apple Silicon configuration. There is no table for M1, M2, or M3, and no guidance on how much unified memory is needed for a given model size beyond the SSD streaming note about 24 GB Macs. Someone on an 8 GB or 16 GB machine has no published data point to reason from.

The alternative, and the actual difference in approach

The obvious comparison is llama.cpp, which also targets local inference on Apple Silicon and also exposes an OpenAI-compatible server through llama-server. The difference is the runtime and the model format. llama.cpp is C/C++ with its own GGUF quantization scheme and its own Metal backend, and it runs on Apple Silicon, x86, and CUDA. SwiftLM is Swift and MLX, which means it is Apple-only by construction and its model weights come from the MLX community on Hugging Face rather than the GGUF ecosystem.

That difference has practical consequences. If a model is published as GGUF and not as an MLX conversion, llama.cpp can run it and SwiftLM cannot. Conversely, MLX quantization schemes such as the 4-bit and 8-bit variants in these benchmarks are native to the MLX toolchain, and TurboQuant is a SwiftLM feature rather than something you get from the GGUF path. The choice is less about raw speed and more about which weight repository your model lives in. The README's own reproduction command uses an mlx-community model identifier, which tells you where the project expects its weights to come from.

Maintenance load, release cadence and the MIT licence

The recent releases are b709, b710, and b711, dated within about a week of each other at the end of August and the start of September 2026. That is a fast build cadence, and the version scheme is a build number rather than a semantic version. For an operator, that means upgrades are frequent and there is no documented compatibility contract between builds. The README does not describe a changelog, a deprecation policy, or which builds are considered stable.

The build path adds its own cost. Building from source initializes git submodules, installs cmake through Homebrew, and compiles Metal kernels before the Swift build. That is a heavier toolchain than a single-language project, and it means the build machine needs the Metal toolchain and Homebrew present. The prebuilt tarball sidesteps this, at the cost of trusting a binary that bundles mlx.metallib.

The licence is MIT, which is permissive and places few obligations on redistribution. That is a statement about the licence text, not legal advice, and it says nothing about the licences of the MLX models you download at runtime, which are governed by their own model cards on Hugging Face. If you ship a product built on SwiftLM, the model licence is the one to check, not the server's.

Who should pick this up, and the first thing to test

SwiftLM fits an Apple Silicon deployment where the models you want are already published in MLX form, where you want an HTTP endpoint rather than a Python process, and where you are prepared to tune flags per model. The published data supports TurboQuant as the default first experiment, since it is the only change that improves throughput and memory together across the tables shown.

It does not fit anyone whose model exists only as GGUF, anyone on non-Apple hardware, anyone who needs a documented guarantee about which OpenAI endpoints are implemented, or anyone on a 24 GB Mac expecting long-context generation at interactive speed. The SSD streaming rows in the README put that configuration at 9 to 11 tok/s, and the TurboQuant combination at 1.6 tok/s at 100K context.

The first thing to verify is not throughput. It is whether your client's request shape survives the compatibility layer, because the README asserts OpenAI compatibility without enumerating it. Point your existing OpenAI SDK at --port 5413 with your real prompt template and your real tool definitions, and see what comes back before you spend time on the benchmark script.

Editorial conclusion

Adopt SwiftLM if you are serving MLX models on Apple Silicon and want an OpenAI-compatible endpoint without a Python runtime, and treat TurboQuant as the option to evaluate first since it is the only change that improves both speed and memory in the published tables. Do not adopt it for a 24 GB Mac expecting usable long-context throughput: SSD streaming is listed at 9.0 to 11.4 tok/s, and SSD plus TurboQuant falls to 1.6 tok/s at 100K context. Before committing, verify three things on your own hardware: that your model's tokenizer and chat template survive the OpenAI-compatible path, that your client tolerates the non-standard --stream-experts and TurboQuant flags living outside the API surface, and that the b7xx release cadence matches your maintenance capacity, because the benchmark tables show the recommended flag combination changing with model bit width.

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. SharpAI/SwiftLM on GitHub