vLLM-Moet: 2-bit MoE experts and an FP4 delta cache for SM120

The problem: checkpoints that do not fit the cards people own

The README frames the target as frontier Mixture-of-Experts models running on consumer and workstation Blackwell hardware: RTX PRO 6000 and RTX 5090. It names three checkpoints, GLM-5.2 at 753B, DeepSeek-V4-Flash at 159B and Kimi-K2.7-Code at 1T, and states plainly that their official checkpoints cannot fit on that hardware. That is the whole premise. This is not a general-purpose inference server. It is a set of changes aimed at one hardware class and one model family, written by someone who wanted those specific weights to answer tokens on cards they could buy.

The audience is correspondingly narrow. You need SM120 silicon, a vLLM tree at exactly v0.24.0, and enough patience to load a 433 GB checkpoint. The README also claims vLLM v0.24.0 is broken as shipped on SM120, which is why the patch exists at all. If you are on Hopper or on an older Ada card, none of this applies to you. If you are on a datacenter Blackwell part, the memory pressure that motivates the 2-bit path is less acute, though the KV cache and speculative decoding pieces might still interest you.

Two bits on the experts, FP4 on the hot path

The compression targets routed experts only. The dense stack keeps whatever precision the checkpoint shipped with, which the README identifies as FP8 for DeepSeek-V4-Flash and NVFP4 for GLM-5.2. The loader re-quantizes modelopt NVFP4 experts (e2m1 times e4m3 block-16 times a per-tensor scale_2) into 2-bit planes at load time, and the README claims this is f64-exact against the reference pipeline on real shards.

The interesting part is why naive 2-bit fails. The README reports that a straight optimal-L2 codebook sends these models into degenerate loops, and attributes it to sign asymmetry rather than error magnitude: the codebook drops one sign's tail, and the per-expert bias accumulates across dozens of layers. Forcing a sign-symmetric codebook of minus four, minus one, one and four at the same L2 error is reported to fix it outright, with 33,023 of 33,024 DeepSeek-V4-Flash tensors selecting it and MTP acceptance landing at or above the FP4 experts (2.73 against 2.68 in the QUANT_PROBE study). The README says the finding reproduces on GLM-5.2 across a 180-tensor sweep.

Recovery is deliberately partial. Decode is described as HBM-bound, and an FP4 read costs twice the bytes of a 2-bit read, so 2-bit stays the default and FP4 is applied where it changes the answer. A delta cache holds hot experts at FP4 with background promote and evict, sized automatically from post-KV VRAM via VLLM_MOE_W2_DELTA_GB=auto. A confidence gate, enabled with VLLM_MOE_W2_GATE=1, re-runs low-confidence steps at FP4: it force-promotes the step's routed experts, replays the graph once, and re-decides. The README says this works inline on tensor-parallel and single-GPU setups, including MTP verify steps, and becomes a full-pipeline replay under pipeline parallelism. The threshold is tunable at runtime.

The kernels behind this are moe_w2_mm, a 2-bit MoE GEMM that decodes a PRMT-LUT in registers and feeds QMMA.SF block-scaled tensor cores at four CTAs per SM, and moe_w4_mm, the FP4 delta GEMM. They ship as sources plus prebuilt cubins for K values of 6144, 4096, 2048, 1024 and 512, which is what makes TP2 and TP4 work without a build step.

Tiering experts across VRAM, pinned RAM and an NVMe pack file

When the 2-bit base still does not fit, the README moves it outward. The first tier is pinned host RAM. The second is an NVMe pack file backed by a pinned-RAM arena, with the GPU acting as an expert cache: a miss triggers a batched fetch and a bit-identical graph replay. The README states that the packs double as a persistent quantization cache, so a reboot skips the re-quantization. The README reports booting from existing packs at about seven minutes against roughly eleven for a full re-quantizing load, which also stages around 405 GiB of transients.

The two-card GLM-5.2 configuration is the clearest illustration of the cost. The 2-bit planes alone are about 190 GiB, which the README says matches the entire two-GPU VRAM budget. The three tiers break down as an NVMe pack plus a 57 GiB per-rank pinned arena for the 2-bit base, a 46 GiB per-rank GPU expert cache, and a small gate-filled FP4 pool for precision. Expert stores need about 136 GiB of host RAM instead of roughly 568 GiB pinned. That is the trade: you spend host memory and NVMe bandwidth to buy back a model that otherwise would not start.

Quality on the bare 2-bit path is not free either. The README admits to artifacts, quoting a wrong capital-of-Poland answer and garbled Polish, and says the FP4 tier corrects them. That is an honest disclosure and it matters for planning: the gate is not a performance knob, it is part of the correctness story on this configuration.

The serving base: KV cache, MTP and parser plumbing

Beyond the expert compression, the patch carries a rebuilt serving base. The README claims vLLM v0.24.0 actually working on SM120, MTP speculative decoding including under pipeline parallelism and described as bit-deterministic, an NVFP4 KV cache at a packed 352 bytes per token, and agent-ready tool and reasoning parsing.

The KV cache number is worth isolating. The README compares 352 bytes per token for nvfp4 against 656 bytes for fp8_ds_mla, which it reports as a 38 percent larger KV pool, 415K tokens becoming 571K at equal settings, at what it calls decode parity. Alternatively the freed VRAM goes to the FP4 pool; the standing four-card configuration is described as running a 19.6 GiB per GPU pool alongside a 175K-token KV.

The MTP claims are the ones I would scrutinise hardest. The README reports DS4 on four RTX 5090 cards under PP4 at 184 tok/s against 93 without, roughly double, and states that greedy decode under pipeline parallelism is bit-deterministic across six identical runs, with and without MTP. It also reports MTP acceptance around 2.6 tokens per step across configurations. These are specific, falsifiable numbers with a methodology document referenced at docs/v024-port.md, which is the right way to present them, but they are self-reported and I have not reproduced them.

Getting it running: the flags the README names

The README does not provide a step-by-step install walkthrough in the excerpt available, so treat the following as the configuration surface it documents rather than a complete procedure. It describes the project as official vLLM v0.24.0 plus a generated runtime patch, which means the patch is applied to a vLLM checkout at that version rather than installed as a standalone package.

The configuration keys named in the README are: VLLM_MOE_W2_DELTA_GB=auto to size the FP4 delta cache from post-KV VRAM, VLLM_MOE_W2_GATE=1 to enable the confidence gate, and --kv-cache-dtype nvfp4 to select the packed KV cache. The two-card GLM-5.2 configuration is described with --max-model-len 131072 and 8 GiB per rank of KV, which the README says measured 157K tokens of KV and passed needle retrieval four out of four at 36K, 86K and 121K prompt tokens on fp8 KV with tolerance zero.

Two operational facts are worth extracting. First, the prebuilt cubins cover K values of 6144, 4096, 2048, 1024 and 512, so your tensor-parallel degree needs to land on one of those or you are building SASS yourself. Second, the README mentions parsers named glm47 for tool calling and glm45 for reasoning, and says the endpoint drives the opencode coding agent directly. If your client depends on a specific tool-call schema, that parser name is the thing to check.

Where this is the wrong tool

The most concrete limitation is that the three-tier path is not a general serving mode. It requires roughly 140 GiB of host RAM for GLM-5.2 on two cards, and the README's own numbers show the cost in throughput: 28 to 32 tok/s on two cards against 105 tok/s for the four-card 2-bit configuration with MTP k=2. The two-card result is an achievement in the sense that the model runs at all, and a poor fit for anything latency-sensitive or multi-tenant.

Context is the second boundary. The README states that GLM's nominal 1M window is KV-bound on four cards, with needle retrieval passing to 126K on the nvfp4 KV cache and to 276K on fp8, and 331K fitting at utilization 0.95. On the two-card configuration the served window is 128K. On DeepSeek-V4-Flash the served windows are 24K on a single PRO 6000, 16K on four 5090s and 32K on a single 5090. If your workload needs the advertised context length of these models, this project does not deliver it.

There is also a correctness caveat the README states itself: bare 2-bit quality produces artifacts, and the FP4 tier is what corrects them. Anyone running with the gate disabled on the two-card path should expect those artifacts. And because the patch is generated against vLLM v0.24.0 specifically, any upstream change to the MoE dispatch path is a rebase you own.

What you would use instead, and how it differs

The obvious alternative is stock vLLM with the official NVFP4 checkpoints, unmodified. That path keeps the checkpoint's native precision on the experts, requires no SASS kernels, and is supported by the upstream project. Its difference is not quality, it is fit: the README's premise is that the official checkpoints cannot be loaded on the target cards at all, so stock vLLM is not a slower version of this, it is a version that does not start on two RTX PRO 6000 cards for GLM-5.2. If you have enough VRAM for the checkpoint as shipped, the case for vLLM-Moet largely evaporates, and you should take the upstream path.

A second comparison is against quantizing the whole model rather than only the routed experts. The README is explicit that the dense stack keeps the checkpoint's precision, FP8 on DeepSeek-V4-Flash and NVFP4 on GLM-5.2, while only the routed experts go to 2 bits. That selective approach is what makes the FP4 delta cache meaningful: there is a precision tier to promote into, and the gate has something to re-run against. A uniform low-bit quantization has no such recovery path, which is the design difference worth understanding before choosing.

Maintenance, licence and what to verify

The repository is Apache-2.0 and not archived, which is permissive and permits commercial use, but I am not a lawyer and this is not legal advice; check the licence text and any third-party notices yourself, particularly around the NVIDIA-derived kernel sources and the modelopt checkpoint format. The README states the pack file doubles as a persistent quantization cache, which means the packs are derived artifacts you will want to manage and version separately from the weights.

Maintenance cost is the real question. There are no retrieved releases, so there is no tagged version to pin and no changelog to read. The patch is described as generated against vLLM v0.24.0, which means the upgrade path is a rebase against a moving upstream, and the hand-written SASS for SM120 is architecture-specific work that will need revisiting on the next silicon. The prebuilt cubins are enumerated for five K values, so an unusual tensor-parallel degree is a build problem, not a config change.

The README points to docs/v024-port.md for methodology. That document, plus the QUANT_PROBE study it references for the sign-symmetry finding, is where I would start before trusting any of the throughput figures above. The numbers are specific and internally consistent, which is a good sign, but they are the author's measurements on the author's hardware, and the sign-bias claim in particular is the kind of result that deserves independent reproduction before you build a deployment around it.

Editorial conclusion

Adopt vLLM-Moet only if you already hold consumer or workstation Blackwell cards and want to serve a frontier MoE checkpoint that does not fit in their combined VRAM; the README's own numbers put 753B GLM-5.2 on two RTX PRO 6000 cards at 28 to 32 tok/s and 159B DeepSeek-V4-Flash on a single RTX 5090 at roughly 31 tok/s. Do not adopt it if you need a stable tagged release, a documented upgrade path, or a multi-tenant SLO: there are no releases, the patch is generated against one vLLM version, and the three-tier configuration depends on roughly 140 GiB of host RAM and an NVMe pack file. Before committing, verify four things in your own environment: that your card is actually SM120, that your vLLM tree is exactly v0.24.0, that the prebuilt cubins cover the K value your tensor-parallel degree produces, and that the NVFP4 KV cache and the GLM tool and reasoning parsers behave as the README describes on your prompts.

Official sources

1. Issues
2. kacper-daftcode/vLLM-Moet on GitHub
3. License: Apache-2.0
4. README