GLM-5.3-Flash-EXL3-2x-DGX-Sparks: serving GLM-5.3 Flash EXL3 on two DGX Sparks
GLM-5.3 Flash EXL3 for 2-4x DGX Sparks
At a glance
- What is it?
- A vLLM overlay that serves a 4 bpw EXL3 build of GLM-5.3-Flash across two NVIDIA GB10 nodes. The interesting part is not the model: it is the NoPE MLA geometry workaround that the stock image cannot do on SM121.
- Who is it for?
- Adopt this if you already own two or more GB10 DGX Sparks, your interconnect is CX7 and you are willing to read the launcher scripts before booting. Do not adopt it if you want a single-node deployment, a pip-installable package, or a model whose weights you can fetch without a 164 GiB download and a Hub mirror.
- Can I use it commercially?
- Yes, with strict conditions. AGPL-3.0 is a network copyleft licence: if people use a modified version over a network, for example as a hosted service, you must offer them its source code under the same licence.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What problem the GLM-5.3-Flash-EXL3 overlay actually solves
GLM-5.3-Flash is a NoPE MLA model. On SM12x hardware the only sparse-MLA backend available is FLASHINFER_MLA_SPARSE_SM120, and the Dockerfile in this repository explains at length why that backend rejects the checkpoint. The packed fp8_ds_mla record is 656 bytes: 512 for the NoPE latent, 16 for scales, 128 for RoPE. Two guards fire. First, concat_and_cache_mla in cache_kernels.cu:866 raises "pe_dim must be 64 for fp8_ds_mla". Second, flashinfer/mla/_core.py:583 requires kv_lora_rank=512, qk_rope_head_dim=64 and a query head dim of 576 for the SM120 sparse entry.
The FlashInfer SM120 sparse-MLA implementations instantiate three model types, according to the Dockerfile comments: DSV3_2 and GLM_NSA at d_qk=576 with 656-byte records and topk in {128, 512, 1024, 2048}, and DSV4 at d_qke=512 with a 584-byte record and topk in {128, 512, 1024} only. GLM-5.3-Flash wants d_qk=512 with index_topk=2048. No kernel instantiates that combination, and the uniform 512-dimensional latent does not fit the DSV4 448/64 split.
The overlay's answer is to run the model in the GLM_NSA geometry that the GLM-5.2 line already uses: zero-pad the 512-dimensional latent to 576. A zero RoPE block contributes nothing to the QK dot product, softmax is unchanged, and the value comes from the 512 NoPE region, so the authors state the result is exact. The cost is stated plainly in the Dockerfile: 656 bytes per token instead of roughly 528, about 24 percent more DSA KV across 11 DSA layers, and no kernel work. That is the trade this project makes, and it is a deliberate one rather than a hidden one.
Who is it for? People who own two or more NVIDIA GB10 DGX Spark machines, want to serve a 4 bpw GLM-5.3-Flash behind an OpenAI-compatible endpoint, and are comfortable with a recipe repository rather than a package. The README says the served model id is GLM-5.3-Flash-EXL3, and the target is a 2x NVIDIA GB10 kit with tensor-parallel size 2 over CX7.
How the EXL3 MoE path and the DFlash2 speculator fit together
Two mechanisms dominate this repository: the grouped MoE kernel and the draft model.
The MoE change is documented as EXL3_FAT_GROUPED=1, the launcher default since 2026-09-07. The README states it replaces the E2 per-expert host loop for "fat" experts with three GPU-driven launches per MoE layer: gather, gate/up plus SwiGLU, and down plus scatter, built from device-side segment tables. The stated properties are no per-expert launches, no weight repacking and no host sync. The implementation lives in overlay/exl3_fat_moe.cu. An isolated layer benchmark in the README reports MoE layer 77 to 91 ms under E2 against 31 ms under E3 at 7168 tokens, and describes the E3 result as numerically indistinguishable from E2 against the LinearEXL3 reference. Decode is stated to be unchanged because E3 never runs on decode-sized steps.
The speculator is DFlash2 at k=7, from incoai/GLM-5.3-Flash-DFlash2. The README is explicit that draft attention must be FLASH_ATTN and that pinning TRITON_ATTN is wrong here, because on this image that mask is causal inside the draft block and collapses accept at later positions. This is the kind of note that only exists because someone hit it.
Weights come from Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw, described as a byte-identical public mirror of brandonmusic/GLM-5.3-Flash-tr3-4bpw at snapshot 5ab363a8, uniform-K4 EXL3/TR3 routed experts, 4 bpw, about 164 GiB across 120 shards. The mirror exists so the recipe stays fetchable if the upstream Hub id moves. KV stays packed fp8_ds_mla. The README warns against passing --moe-backend marlin, since that flag is the NVFP4 FLASHINFER_CUTLASS workaround and the wrong backend for an EXL3 checkpoint.
Note the divergence the README itself flags: the brandonmusic Hub card describes a TP2/EP2/DCP2 setup with calibrated NVFP4 MLA KV on the SM120 B12X image, which is not this overlay. Two recipes, two images, similar names.
Installing the overlay and serving a first request on two Sparks
There is no package to install. The repository is a Docker-based recipe with shell launchers, and the entry point is start.sh, which on first run copies .env.example to .env for you. The README states the caller's exports override the file, including explicit empties, so KEY= ./start.sh clears a value.
Start by editing the node section of .env. The example file gives these keys:
HEAD_IP=10.0.0.1
WORKER_IP=10.0.0.2
HEAD_CX7_IF=enp1s0f1np1
WORKER_CX7_IF=enp1s0f0np0
HEAD_CX7_IB=rocep1s0f1
WORKER_CX7_IB=rocep1s0f0The comments in .env.example are worth reading before you change anything. NCCL cannot use the 10.0.0.x loopback aliases, and without the interface pins ncclCommInitRank hangs. The device names are per kit: the file notes that other Spark pairs may show names like enP2p1s0f1np1 or roceP2p1s0f1, and suggests listing yours with ls /sys/class/infiniband and ip -br a. Exporting generic NCCL_SOCKET_IFNAME or NCCL_IB_HCA does not override the per-node values the containers receive.
The GID index is the part most likely to bite. The file states the RoCEv2 GID index is per-NIC and each node needs the index carrying its own ::ffff:<ip> entry. An all-zero entry kills that rank about 60 seconds in with ibv_modify_qp errno 61. You can inspect the table with this command from the comments:
cat /sys/class/infiniband/<dev>/ports/1/gids/{0..7}If one index works on both nodes, set NCCL_IB_GID_INDEX. If they differ, set HEAD_GID and WORKER_GID per rank instead. Preflight checks every listed device on each node.
Weights are fetched with download.sh, which the top-level listing shows alongside the launchers. Once the service is up, the README states the API listens on port 8888 and the served model id is GLM-5.3-Flash-EXL3, so an OpenAI-compatible client pointed at http://<head>:8888/v1 with that model name is the first thing to try. The README does not document a health endpoint or a readiness probe, so poll the models list rather than assuming a port that accepts connections is ready to prefill.
For three Sparks there is a sibling launcher, start-tp3.sh, on the same image and weights, with its own .env.tp3.example. A four-node variant, start-tp4.sh and .env.tp4.example, also exists in the top-level listing. The README does not state whether the 3x and 4x paths carry the same measured numbers as the 2x kit.
Where this recipe breaks: context, host memory and the wrong hardware
The README is unusually candid about limits, and they are the reason to read it before downloading 164 GiB.
Context headroom is the first constraint. The shipped default is now 850k context at GPU_MEM_UTIL=0.85 with GLM53_INDEXER_WORKSPACE=rightsize, which the README says boots with about 0.5 GiB of KV margin. A 256k prefill ran at 900k context and util 0.87 with driver retries but no failure. But E3 keeps a 560 MiB fat-row scratch that vLLM's profile run charges to the KV budget, so at 1M context and util 0.87 the pool no longer fits one 1M request on this kit, which the README says needs 14.52 GiB. 500k needs 10.98 GiB and boots reliably at 0.84. If you were planning on the full 1M window, plan on a lower util setting and a smaller pool.
Host memory is the second constraint, and it is worse. The README states that prompts at or above roughly 100k are near the head node's host-memory limit at any setting, and that a 256k prefill at util 0.87 with zero MemAvailable crashed the head on 2026-09-06. The 256k row in the results table was run at 0.84. This is a failure mode that takes down the serving process, not a slow path.
Hardware mismatch is the third. This is an SM121 GB10 recipe with native sm_121a cubins and CX7 interconnect assumptions. The Dockerfile says the stock image cannot serve this checkpoint on GB10, which is the whole reason the overlay exists. If you are on a different accelerator, or on a single Spark, the recipe does not apply. The README does not document a single-node path.
Finally, the numbers are kit-specific. The prefill table is labelled with a date and a specific image tag, glm53-flash-sm121:e3-20260907, and the E2 comparison comes from named log directories. The README does not claim these transfer to other Spark pairs, and the interconnect section implies they may not.
The alternative: the upstream brandonmusic recipe on the SM120 B12X image
The README points at a real alternative rather than pretending there is none. The Hub card on brandonmusic describes a TP2/EP2/DCP2 configuration with calibrated NVFP4 MLA KV, running on the SM120 B12X image tagged verdictai/glm53-flash-exl3-k4 with a v84-dflash2 suffix. The README states directly that this is not the overlay.
The difference in approach is the KV cache and the GEMM path. This repository keeps packed fp8_ds_mla KV and EXL3 weights, and treats the MLA geometry as the thing to fix, which is why it zero-pads the latent to 576 and pays about 24 percent more DSA KV across 11 DSA layers. The B12X route uses calibrated NVFP4 MLA KV instead, which changes what the calibration step costs you and what the KV pool budget looks like. The README also notes that --moe-backend marlin belongs to the NVFP4 FLASHINFER_CUTLASS workaround on that side, and passing it here is wrong.
Which one to pick depends on what you are optimising. If you want the EXL3 checkpoint and the fp8_ds_mla record, this overlay is the one that addresses the SM121 kernel gap. If you want the NVFP4 KV path with its calibration, the upstream card is the reference. The README does not present a head-to-head benchmark between the two, so do not read the prefill table as a comparison against B12X. It is a comparison against this project's own earlier E2 kernel.
Licence, maintenance and what an upgrade costs you
The repository is AGPL-3.0, and the top-level listing also carries a LICENSE.MIT file. Two licence files in one tree means you need to check which one covers what before you redistribute anything; the README does not explain the split. AGPL-3.0 is a network-copyleft licence, so if you expose a modified version of this overlay as a service, the obligations attach to the modified source. That is a statement about the licence text, not legal advice, and the weights carry their own terms on the Hub, separate from this repository.
On maintenance: the last push was on 2026-09-16, one day before this writing. There are no retrieved releases, so there is no tagged version to pin against. If you deploy this, pin the image tag yourself. The README already does this in its own receipts, naming glm53-flash-sm121:e3-20260907 for the prefill run, which is the pattern to copy.
Upgrade cost is mostly the .env surface. The launcher defaults have moved before, as the EXL3_FAT_GROUPED default change on 2026-09-07 shows, and an image that changes kernel behaviour can change your KV budget because of scratch allocations the profile run charges to you. The README records exactly this: E3's 560 MiB fat-row scratch is why 1M context at util 0.87 stopped fitting. A kernel upgrade is therefore also a capacity change. Re-read the context and headroom section of the README after any image bump, and re-check the RoCEv2 GID indices if you change NICs or re-cable. The README does not document a rollback procedure, so keep the previous image tag available.
Editorial conclusion
Adopt this if you already own two or more GB10 DGX Sparks, your interconnect is CX7 and you are willing to read the launcher scripts before booting. Do not adopt it if you want a single-node deployment, a pip-installable package, or a model whose weights you can fetch without a 164 GiB download and a Hub mirror. Before you start, verify three things: that your RoCEv2 GID index is populated on every listed device (the .env.example warns an all-zero entry kills a rank about 60 seconds in), that the KV pool still fits your intended context (the README states 1M context at util 0.87 no longer fits one request on this kit), and that your head node has host memory to spare, because a 256k prefill at util 0.87 with zero MemAvailable crashed the head on 2026-09-06.
Frequently asked questions
Can you run a GLM 5.2 on DGX Spark?
This repository is about GLM-5.3-Flash, not GLM-5.2, but its Dockerfile notes that the GLM-5.2 line already uses the GLM_NSA geometry, and the overlay borrows that geometry by zero-padding the 512-dimensional latent to 576. The README does not document a GLM-5.2 deployment recipe of its own.
What models can you run on a DGX Spark with this overlay?
The served model id is GLM-5.3-Flash-EXL3, backed by the EXL3/TR3 4 bpw checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw at roughly 164 GiB across 120 shards. The overlay is built for the NoPE sparse MLA geometry of that checkpoint on SM121, so it is not a general-purpose model runner.
Is DGX Spark unified memory a problem for GLM-5.3-Flash-EXL3?
The README ties context headroom to memory budgets rather than describing the memory architecture. It states that at 1M context and util 0.87 the KV pool no longer fits one 1M request on this kit, and that prompts at or above roughly 100k are near the head node's host-memory limit at any setting.
How many DGX Sparks can be linked together for this recipe?
The repository ships launchers for two, three and four nodes: start.sh with .env.example, start-tp3.sh with .env.tp3.example, and start-tp4.sh with .env.tp4.example. The measured prefill and decode numbers in the README come from the 2x GB10 kit with tensor-parallel size 2 over CX7.
Community notes