slotstream: streaming Qwen3.8-Flash-Next experts from SSD on Apple Silicon

What slotstream solves, and who has the problem

A 125B mixture-of-experts model quantised to 4-bit still weighs about 104 GB. That does not fit in the unified memory of any consumer Mac, and the usual alternatives are renting GPU time or accepting a much smaller model. slotstream takes a third route: the weights stay on SSD, and only the experts a token actually routes to are pulled into memory. The README's headline is blunt about the target, "Run a 105 GB AI model on a 48 GB Mac."

The audience is narrow and specific. You need Apple Silicon, macOS 14 or later, at least 16 GB of memory and about 110 GB of free SSD space. The installer has been tested on macOS 14 and 15, while model runs have been tested on macOS 26. On an 8 GB Mac the README says even the smallest memory plan does not fit, so slotstream refuses to start rather than thrash swap. That refusal is a design decision worth noting: a tool that silently degraded into swap would produce a worse experience than one that exits.

Two groups get something out of this. Mac owners who want a large open model running offline, with no Python and no cloud API account. And developers who want to point existing tooling at a local endpoint, since slotstream exposes an Ollama-compatible API and a Swift library.

How expert streaming and decode lookahead actually work

The mechanism is SSD-backed paging of expert weights. A mixture-of-experts layer routes each token to a small subset of experts, so the full 104 GB does not need to be resident. slotstream keeps the non-expert parts of the model in memory and reads expert blocks from disk on demand. After a one-time download the model works offline.

The obvious cost is that disk reads sit on the critical path. Version 0.2.16 added decode lookahead, which the README describes as predicting which parts of the model the next layers will need and reading them from the SSD before they are requested. The same change keeps a faster copy of the routing weights and waits for the GPU less often. The README reports the lookahead measured 1.11x faster decode, moving its development Mac from 11.79 to 13.47 tok/s at a 20 GB memory target on prompts it was never tuned on, with identical output and speculative decoding in both configurations. Those are pre-release numbers for the shipped configuration, and the README says the latest published release determines what the installer downloads.

Memory behaviour is tiered rather than fixed. Auto mode picks the memory target, cache size, speculative decoding and context window, taking the largest of 32,768, 65,536, 131,072 or 262,144 tokens that keeps speculative decoding on and keeps one complete conversation ready for follow-up turns. A larger manually selected memory target can improve speed, which is why the M5 Max rows in the measured table climb from roughly 21-22 tok/s at a 34.6 GB auto target to about 31.5 tok/s at 73 GB.

Installing slotstream and running a first prompt

The repository ships install.sh at the top level, and the README's Get started link points at the install section. The installer downloads the model, so budget the roughly 110 GB of free SSD before you begin. Check your chip and memory first through the Apple menu, About This Mac.

./install.sh

After the one-time download the model runs offline. The README states no Python and no cloud API account are required, and that the CLI, APIs and Swift library remain independently usable even though the author is building a separate app, Sevra, on top of the engine.

If you are building from source rather than using the installer, the Makefile is the entry point. It notes that SwiftPM cannot compile the Metal shaders because of an mlx-swift limitation, so a prebuilt metallib matching the vendored MLX version 0.31.1 is fetched and colocated with the binary.

make build
make checks

The build target runs swift build -c release and then copies the metallib next to the release binary as mlx.metallib. The checks target builds the debug binary and runs the check catalogue at tier t0, which the Makefile says needs nothing beyond the build. Tier t1 and above touch MLX. The Makefile also notes that swift test needs Xcode, because Command Line Tools ship no XCTest, which is why the catalogue is a standalone executable rather than an XCTest bundle. That is a real constraint on contributors who only have the command line tools installed.

For application integration, the README points developers at the APIs and the Swift library. The Ollama-compatible API means existing clients that speak that protocol can be pointed at the local server; the README does not document the exact port or endpoint paths, so check the repository before wiring anything up.

Where slotstream is the wrong choice

The throughput numbers are the first limitation. The measured table spans 1.41 tok/s on a 16 GB M2 Mac mini up to about 31.5 tok/s on a 128 GB M5 Max with a 73 GB manual target. Even the fast end is conversational, not batch. Anyone expecting to embed this in a latency-sensitive pipeline should read those numbers as the ceiling of what the hardware allows, not as a floor.

The README is unusually candid about the planning ranges. The upper ends of the High and Ultra estimates assume an M5 Max-class chip, a fast internal SSD and a larger manually selected memory target, and it states plainly that the High upper estimate transfers the 128 GB M5 Max's 48 GB target result to a comparable Mac that has not been benchmarked. Ultra comes from an auto-to-73 GB target sweep on the same machine. The README calls these estimates, not measured limits or statistical confidence intervals. The two M5 Pro rows in the table also change both software and settings, so they do not isolate the effect of memory.

Platform coverage is the second boundary. Windows, Linux and Intel Macs are not currently supported. An 8 GB Mac is explicitly out, with the README saying support is coming soon for that tier. And the whole approach trades RAM for disk: a slow internal SSD will land you below the published ranges, because expert reads sit directly on the decode path. If your workload needs the full model resident and fast, this is the wrong architecture, and a smaller dense model that fits entirely in memory will beat it.

How slotstream differs from llama.cpp and Ollama's own runner

llama.cpp and the runner inside Ollama both load quantised weights into memory, with optional mmap so the operating system can page file-backed pages. mmap is a general mechanism: the kernel decides what stays resident, and it has no notion of which expert the next token will route to. slotstream's decode lookahead is the difference. It predicts the next layers' expert needs and issues the SSD reads before they are requested, and it keeps a faster copy of the routing weights. That is model-aware prefetching rather than relying on the page cache.

The second difference is the interface. Ollama's own server is the reference implementation of the protocol; slotstream implements an Ollama-compatible API so existing clients can talk to it. If your tooling already targets Ollama, the migration is a base URL change rather than a rewrite, though the README does not document the port or endpoint list, so verify against the repository.

The third is language and packaging. slotstream is Swift with MLX and ships as a native macOS binary plus a Swift library, with a Makefile that fetches a metallib from the mlx-metal PyPI wheel because SwiftPM cannot compile the Metal shaders. llama.cpp is C and C++ with a wide platform surface. If you need Linux or Windows, slotstream is not in the running at all, and that is a hard boundary rather than a maturity gap.

Maintenance, releases and what the licence permits

The last push to the default branch was on 2026-09-15. Releases are frequent and small: v0.2.16, v0.2.17 and v0.2.18 all landed between 2026-09-13 and 2026-09-14. The repository is not archived. That cadence matters here because the README ties behaviour to versions, stating that the latest published release determines what the installer downloads, so an installer run today will not necessarily fetch the configuration described in the 0.2.16 benchmark paragraph.

Upgrade cost is mostly disk and bandwidth. Each release that changes the model or the memory plans can force a fresh download against that roughly 110 GB budget. The repository carries a CHANGELOG.md and a MEASUREMENTS.md, and the README points at docs/HARDWARE.md for planning ranges, test conditions and credits. If you are tracking performance across releases, those two files are where the numbers and their caveats live.

The licence is MIT. That permits commercial use, modification and redistribution provided the copyright notice and permission notice are included. It offers no patent grant and no warranty, and it says nothing about the model weights you download, which carry their own terms from their publisher. That is a distinction worth checking before shipping anything, though it is not legal advice and the repository's LICENSE file is the authority.

Editorial conclusion

slotstream is for Apple Silicon owners with at least 16 GB of memory and roughly 110 GB of free SSD who want a large MoE model locally without a cloud account, and for developers who want to reach it through an Ollama-compatible API or the Swift library. It is not for 8 GB Macs, Intel Macs, Windows or Linux, and not for anyone who needs predictable latency rather than a warm-reply throughput range. Before committing, check your free SSD space, then read docs/HARDWARE.md for the memory plans and MEASUREMENTS.md for the test conditions behind the published tok/s figures, since the README states the upper ends of the High and Ultra ranges are transferred or extrapolated rather than measured on those Macs.

Frequently asked questions

Official sources

1. carloslfu/slotstream on GitHub
2. License: MIT
3. Project website
4. README
5. Releases