mac-code: running a 35B agent on a 16 GB Mac by streaming FFN weights from SSD

The RAM wall mac-code is trying to move

The problem is narrow and concrete. A quantized 35B mixture-of-experts model is around 19.5 GB at Q4 and 10.6 GB at IQ2_M. A Mac mini M4 with 16 GB of unified memory cannot hold the Q4 file, and once macOS starts swapping, throughput collapses. mac-code's answer is to stop treating the model as one object that must be resident. The README frames the project around the question in its own table: what can you run, on which Mac, at what speed. That table is the product. Everything else in the repository exists to fill in one of its cells. The audience is Apple Silicon owners who want a coding or research agent locally, without a per-token bill, and who are willing to accept a slow path when the fast path does not fit.

Two memory strategies, and the gap between them

The first strategy is compression. The recommended configuration downloads Qwen3.5-35B-A3B in UD-IQ2_M at 10.6 GB and runs it entirely in RAM through llama.cpp, with q4_0 key and value caches and a 12288-token context. The README reports 30 tok/s on a 16 GB Mac mini M4. The second strategy is streaming, and it is where the project's own research lives. Attention weights, embeddings, norms and the KV cache stay pinned in RAM, roughly 4 to 6 GB. The FFN weights, which dominate the parameter count, are loaded layer by layer from SSD, used for one matmul, and discarded. Memory stays flat as generation proceeds. The README gives the per-layer transfer as 165 to 221 MB for dense models. For MoE models the same step loads only the active experts, about 14 MB instead of all 256, which the README cites as the reason MoE streaming is roughly ten times faster than dense streaming. That single distinction explains the whole speed table: 4.3 to 5.4 tok/s for MoE streaming versus 0.15 to 0.18 tok/s for dense.

Getting the 35B path running

The Quick Start is short and specific. Install llama.cpp with Homebrew, install rich and ddgs with pip3, then pull the GGUF with huggingface_hub into $HOME/models. The server command matters more than it looks: llama-server takes --model pointing at the IQ2_M file, --port 8000 --host 127.0.0.1, --flash-attn on, --ctx-size 12288, --cache-type-k q4_0, --cache-type-v q4_0, --n-gpu-layers 99, --reasoning off, -np 1 and -t 4. Then python3 agent.py connects to it. The 9B variant is the same shape with a different model file and --ctx-size 65536, which the README lists as 16 to 20 tok/s on any Mac with 8 GB or more. Note that -t 4 pins four threads. On a machine with performance and efficiency cores, that is a deliberate choice to avoid scheduling across the wrong cluster, and it is worth leaving alone until you have a baseline. The README also references setup.sh as a one-command install and config.example.json as an example configuration, though it does not document the keys inside the config file.

The streaming path is a research tree, not a product

This is the part of the repository that decides whether mac-code is usable for you. The 27B dense streaming route requires installing mlx-lm and transformers, downloading mlx-community/Qwen3.5-27B-4bit with snapshot_download, running split_dense_27b.py once to produce the stream files, then running flash_stream_27b.py. The 35B MoE agent is the same pattern: cd research/flash-streaming and python3 moe_agent.py, but the README states plainly that it requires pre-built stream files and points at that directory for the split and rebuild tools. It does not give a single command that produces those files for the MoE case. That is a real gap for anyone who wants the 22 GB model on a 16 GB machine. The directory is described as a research journey where each file is a step, and the file list is truncated in the README, so the exact entry point for building MoE stream files cannot be confirmed from the supplied material. Treat the streaming path as a working experiment with reproducible measurements, not a supported install.

Disk is the hidden cost, and external NVMe is the escape hatch

Streaming trades RAM for storage. The README's disk table lists processed sizes and free-space requirements: 17 GB and about 20 GB free for the 30B class, 19 GB and about 25 GB for the 35B, 43 GB and about 50 GB for 80B, roughly 65 GB and 70 GB for 122B, and about 130 GB and 140 GB for 235B. Those are not download sizes alone. They are the post-processing footprint after the split step. The README suggests an external NVMe drive for the larger entries, with mlx-sniper download qwen3-next-80b -o /Volumes/MySSD/qwen3-next-80b followed by mlx-sniper run /Volumes/MySSD/qwen3-next-80b -p "hello" -v. That mlx-sniper tool is not described elsewhere in the material, so its relationship to the rest of the repository is unclear. If your internal SSD is nearly full, this project is not a candidate regardless of how much RAM you have.

Where the design breaks down

The documented dense streaming result is 0.18 tok/s on a 16 GB Mac mini M4 for the 27B. That is about one token every five and a half seconds. For a chat agent that runs shell commands and web searches, the round trip through several hundred tokens of reasoning is measured in tens of minutes. The README is upfront that this is slow, and it does not pretend otherwise, but the practical consequence is that the dense streaming path is a demonstration that the method works, not a configuration you would use for daily work. The second failure mode is quality at higher routing bias. The README states that bias=1.0 is the universal safe maximum and that quality degrades at 1.5 on both tested models. If you tune that parameter upward to chase speed, you are outside the range the author validated. Third, the batched union-of-experts prototype at 5.1 tok/s is verification speed, not generation speed, and the README says so directly. It is useful for speculative decoding research and not for interactive use.

How this differs from llama.cpp and Ollama alone

llama.cpp and Ollama both load a model into memory and keep it there. Ollama in particular manages model files, a server and a pull-based model registry, and it will fail or fall back to CPU layers when the model exceeds available memory. mac-code uses llama.cpp as its inference server, so it is not an alternative to it. What it adds is the layer above: an agent loop with /agent, /raw, /search, /stats, /clear and /quit commands, web search through ddgs, shell execution, a dashboard.py monitor, and a web UI under web/. The genuinely different part is the streaming research, which llama.cpp does not do by default in the form described here. The closest comparison in the material is mlx-lm, which mac-code also uses for the dense streaming path. If you already run Ollama and your model fits in RAM, mac-code offers an agent and a monitoring dashboard rather than a new inference engine. If your model does not fit, the streaming path is the reason to look at this repository at all.

Maintenance, licensing and what to check before you commit

The repository has no releases and no homepage, and the licence field is unknown. That last point matters if you intend to use mac-code at work: without a licence file, the default position under copyright is that you have no granted rights to redistribute or modify, and the README does not address it. The last push recorded is 2026-04-09. The dependency surface is small but not trivial: llama.cpp from Homebrew, rich and ddgs from pip, and mlx-lm plus transformers for the streaming path. Each of those moves independently. The model files come from unsloth and mlx-community on Hugging Face, so their availability and quantization naming are outside the project's control. Before adopting, confirm the licence status directly with the repository owner, check that the exact GGUF filenames in the Quick Start still resolve on the Hub, and decide whether the 30 tok/s IQ2_M path or the streaming research is what you actually need. The first is a usable local agent today. The second is a method with published measurements and an unfinished install story.

Editorial conclusion

Adopt mac-code if you have an Apple Silicon Mac with at least 16 GB and you want a local agent with web search and shell access without paying per token. The 35B IQ2_M path is the one to start with: it is the only configuration the README shows running at 30 tok/s on 16 GB. Do not adopt it if you need interactive latency on dense models, since the documented 27B dense result is 0.18 tok/s, roughly one token every five and a half seconds. Before committing, verify three things yourself: that your free disk covers the processed size plus headroom (about 25 GB for the 35B), that llama-server starts with the exact flag set shown in the Quick Start, and that the stream files for the MoE agent can actually be built from research/flash-streaming, because the agent will not run without them.

Official sources

1. Issues
2. README
3. walter-grace/mac-code on GitHub