BigMoeOnEdge: running MoE models larger than RAM on a phone, CPU only

The problem BigMoeOnEdge targets: models that outgrow the device

A Mixture-of-Experts model is built from many small expert subnetworks, and each generated token activates only a few of them. The README states this plainly and then draws the consequence: most of the weights sit idle on any given token. Every mainstream runtime, however, still holds or pages the whole file in memory. On a 12 GB phone, a 20 GB model either refuses to load or thrashes while the operating system swaps.

BigMoeOnEdge is for people who want to run those models anyway, on hardware they already own. The README names three cases. A roughly 60 GB model on a 12 GB phone cannot be resident at all, but streamed it runs at usable speed. An 18 to 22 GB model sits in the worst zone for mmap, where the OS evicts weights as fast as it reads them and other apps get killed; streamed, the README claims up to 5 tok/s with byte-identical output. Even a model that technically fits, an 8B-class MoE on a phone with a few GB free, can run inside a capped cache budget instead of squeezing out everything else. All of this is plain CPU inference: no GPU, no NPU, four cores and the phone's flash storage.

How the streaming path works, and what llama.cpp still does

The design keeps the small always-needed part of the model at hand and reads only the experts each token requests, directly from flash storage at the moment they are needed. The rest of the model stays on disk. That is the whole mechanism, and it is why the output is described as lossless: the engine is not approximating the model, it is feeding the same weights to the same computation, just fetching them later.

The project is built on top of llama.cpp's public API. Quantization formats, tokenizers and chat templates come from llama.cpp itself, so MXFP4 and Q4_K_M stream through the same code path. Adding a new MoE architecture is one row in a registry rather than a change to the streaming path, and following an upstream llama.cpp release is a submodule bump. The README illustrates the payoff with Qwen3.8-Flash-Next, which it says ran on the same phone the day the weights appeared, before upstream support existed. A large part of that model is a lookup table, and the engine never loads it: it leaves the table on storage and reads only the sixteen entries each word needs.

The repository layout matches that description. core/, cli/, tools/, examples/, scripts/, tests/ and third_party/ sit at the top level, with third_party/ holding the llama.cpp submodule referenced by .gitmodules. Multi-shard ggufs are read natively, so the three shard files Hugging Face ships for DeepSeek V4 Flash 0731 need no merge step.

Installing BigMoeOnEdge on Android and running a first model

The README's phone path requires no build. Install the APK from the latest release, open the Get a model card, and tap a catalog entry. The catalog lists Qwen3-30B-A3B (~18.6 GB), Qwen3.6-35B-A3B (~22.3 GB) and Gemma-4-26B-A4B (~17 GB), each past most phones' RAM. The downloader also accepts any direct gguf URL, so any model from a supported architecture family streams the same way.

When the download finishes, pick the model and chat. The telemetry panel shows tok/s and the live compute-versus-flash split, which is the fastest way to see whether streaming is actually happening or the model has become resident. Every streaming knob lives in Settings.

For multi-shard models such as gpt-oss-120b, DeepSeek V4 Flash 0731 and Qwen3.8-Flash-Next, the app fetches the shards one after another under a single progress bar and resumes an interrupted download. The engine reads the split set natively, so there is no merge step. The README does not document a manual shard-merge procedure, because none is needed.

The same engine builds unmodified on desktop, where a model past RAM streams from the SSD out of the box. The README does not give desktop build commands, and the CMakeLists.txt at the repository root and the cli/ directory are where that path lives.

What the numbers do and do not tell you

The headline figures in the README are 0.94 tok/s for DeepSeek V4 Flash 0731 (284B parameters, about 91 GB on disk) and 3.48 tok/s for Qwen3.8-Flash-Next (125B, about 80 GB at Q2_K), both on a 12 GB phone, recorded in the demo app. Treat these as order-of-magnitude expectations from the author's device, not as a specification. Flash speed varies enormously between phones, and the compute-versus-flash split shown in the telemetry panel is the variable that actually determines your throughput.

A sub-1 tok/s rate is readable but not conversational. The README describes 0.94 tok/s as real time, which is accurate as a statement about the recording and misleading as a statement about interactivity. For a 284B model on a phone, waiting several seconds per token is the honest expectation.

The lossless claim is narrower than it sounds. It means the output is byte-identical to running the same model fully resident, which is a statement about the streaming mechanism, not about quantization. The README also says the engine never loads Qwen3.8-Flash-Next's lookup table and reads only the sixteen entries each word needs; that is a claim about that architecture's structure, and it should not be generalised to every model in the supported list.

Where BigMoeOnEdge is the wrong tool

Dense models gain nothing here. The entire design rests on sparse activation, where a token touches a few experts out of many. A dense model activates all of its weights on every token, so there is nothing to skip and streaming degenerates into reading the whole file repeatedly. The README's framing is explicit that the more of a model that is there to be consulted rather than held, the better this engine fits.

If you have a GPU or an NPU and want to use it, this is not the project. The README states plainly that inference is CPU-only, four cores and flash storage. That is a deliberate constraint for phones, but on a desktop with a capable GPU it means leaving the fastest hardware idle.

Latency-sensitive applications are also out of scope. At 0.94 tok/s on the largest model, a 200-token answer takes roughly three and a half minutes. Anything resembling an interactive assistant on a model far past RAM will feel slow, regardless of how stable the streaming is.

Finally, architectures outside the supported families will not load. The registry makes adding one cheap for a maintainer, but a user cannot add a row without rebuilding. The README does not publish a fallback path for unsupported architectures.

How it differs from llama.cpp's mmap path

The obvious alternative is llama.cpp itself, which BigMoeOnEdge builds on. llama.cpp loads a gguf through mmap and lets the operating system decide what stays in page cache. That works well when the model fits comfortably in RAM and degrades sharply when it does not: the OS evicts pages as fast as the model reads them, throughput swings, and on a phone the memory pressure kills background apps. BigMoeOnEdge replaces that implicit policy with an explicit one. It caps the cache, knows which experts a token needs, and fetches exactly those from storage.

The trade-off is real. mmap lets the kernel use every spare byte of page cache for weights, and on a device with enough free memory that is strictly better than a capped cache. BigMoeOnEdge wins precisely in the regime where the model does not fit, and gives up ground where it does. The README acknowledges this by listing models that barely fit as a third case, where the benefit is running inside a budget you choose rather than running faster.

Project-level differences matter too. llama.cpp is general-purpose and supports dense and sparse models across many backends. BigMoeOnEdge is narrower: MoE streaming on CPU, with llama.cpp as a submodule. Its supported-model list, not its feature list, is the thing to check before adopting it.

Maintenance, releases and the Apache-2.0 licence

The repository is not archived, and the last push was on 2026-09-07, which is recent. Releases v0.22.0 and v0.23.0 landed on 2026-08-28 and 2026-08-29, and v0.24.0 on 2026-09-07, so the cadence over that window was roughly weekly. That is a fact about the release history, not a promise about the next one.

The upgrade cost has two parts. Following llama.cpp is described as a routine submodule bump, which is the cheap part. Following new model architectures is a registry row plus whatever the architecture needs from the streaming path, which is the part that can surprise you. The README's Qwen3.8-Flash-Next example shows both: a new architecture arrived as a registry row, but that architecture also had a lookup-table structure the engine had to handle by leaving it on storage.

The project is Apache-2.0. That license permits commercial use, modification and redistribution, and it includes an explicit patent grant and a patent-termination clause. It also requires that you preserve copyright and licence notices and state significant changes. Note that the project depends on llama.cpp through third_party/, and llama.cpp carries its own MIT licence; if you redistribute a binary, both sets of notices apply. This is a description of the licence terms, not legal advice, and the LICENSE file at the repository root is the authoritative text.

Editorial conclusion

Adopt BigMoeOnEdge if you want to run a Mixture-of-Experts model whose file is several times your device's RAM, on a stock Android phone or a desktop with an SSD, and you accept CPU-only speed measured in single-digit tokens per second. Do not adopt it for dense models, for GPU or NPU acceleration, or for latency-sensitive interactive use. Before trusting it, verify that your target architecture appears in the supported list, that your gguf is a single file or a split set the engine reads natively, and that the byte-identical output claim holds for your quantization by comparing a short generation against a fully resident llama.cpp run.

Frequently asked questions

Official sources

1. Helldez/BigMoeOnEdge on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases