Qwen3.8-Flash-Next on Two DGX Sparks: What the vLLM Scripts Actually Do

The problem: one GB10 cannot hold Qwen3.8-Flash-Next at full context

A DGX Spark node has 128 GB of unified memory on a GB10 part (sm_121). The NVFP4 checkpoint of Qwen3.8-Flash-Next is 133 GB across 11 shards, so it does not fit on one node with room left for a KV cache. The repository's answer is to split the model across two nodes with tensor parallelism of 2, add expert parallelism, and run MTP with three speculative tokens. The target reader is someone who already has two Sparks cabled together and wants a working launch rather than a research project. The README is explicit that the numbers it quotes for KV cache budget and default runtime were read from a running server via docker logs and docker inspect, at GPU_MEMORY_UTILIZATION=0.835, which is the value the shipped .env files use. That matters because it means the memory figures are tied to a specific utilization setting, not to a theoretical maximum.

How the two-node launch is assembled

The flow has seven documented steps. download.sh pulls MODEL_ID from .env into the head node's Hugging Face cache. Distribution then copies the checkpoint to the worker, either by rsync over the ConnectX link or, if NFS_SHARE is on, by exporting the head's cache read-only and mounting it on the worker. The image vllm/vllm-openai:qwen38-flash-next is made present on both nodes. Two patches are then extracted from that image and bind-mounted at runtime rather than baked into a new image: ple_layer.py becomes files/ple_layer_patched.py, and modelopt.py becomes files/modelopt_patched.py, the latter routing MXFP8 shapes that FlashInfer's mm_mxfp8 cannot execute to a BF16 emulation kernel. A preflight check refuses to launch if another process holds a GPU on either node, unless REQUIRE_IDLE_GPU=false. Finally the worker starts as rank 1 and the head as rank 0 serving on port 8888. Both launch scripts render the same VLLM_ARGS array, and the head's rendered script is kept as .last_head_launch.sh so you can inspect what was actually passed. The patch-then-bind-mount approach avoids rebuilding the image, but it also means the runtime behaviour depends on the image tag staying the same under you.

Installing it and reaching a first served request

The README's quick start begins by copying the sample environment file and editing it. The fields you must fill are the two node IPs, the inter-node interface, and the InfiniBand HCA. The sample file shows the expected shapes: IFACE looks like enp1s0f0np0, and IB_HCA uses an exact-match form with a leading equals sign.

cp .env.sample .env
vim .env

Downloading weights happens on the head node. The README notes that the stock NVFP4 model is the default MODEL_ID, and that a gated checkpoint requires ABLIT=1 together with an HF_TOKEN and acceptance of the terms.

./download.sh

The launch step syncs to the worker, applies the patches and starts the server. Passing --no-download is the documented form when the weights are already in the head cache.

./start.sh --no-download

One manual step is deliberately left out. start.sh does not need root and does not drop page caches; the README tells you to do it on both nodes before a launch.

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Roughly eleven minutes after launch, the README says to confirm the KV cache pool that vLLM actually allocated by grepping the container log. If the grep returns nothing, the server has not reached the allocation phase yet.

docker logs vllm-fn 2>&1 | grep -E "Available KV cache memory|GPU KV cache size"

The FP8 checkpoint trades context for a different weight format

There is a second launch path. start-fp8.sh serves the official Qwen/Qwen3.8-Flash-Next-FP8 checkpoint instead of NVFP4, with cluster IPs, TP and EP settings, image and ports still coming from .env. The README states the context is native 262K with YaRN off, because FP8 weights leave too little KV for 1M on this hardware, and that available KV cache is around 500k tokens rather than the 3.65M of the nvidia NVFP4 checkpoint with KV_CACHE_DTYPE=fp8. That is a large difference in how many tokens can be resident, and it is the kind of number worth checking against your workload before choosing a checkpoint. The README also draws two distinctions that are easy to miss: this is not FP8_DENSE=true, which it describes as hybrid NVFP4 experts with FP8 dense projections, and it is not fp8 KV on the stock NVFP4 model. Both launch scripts create a container named vllm-fn on the same port, so you must run ./stop.sh before switching between them. The API name under FP8 is qwen3.8-flash-next-fp8, and ABLIT=1 is ignored once start-fp8.sh sets OVERRIDE_MODEL_ID.

Where the setup will bite you

The worker's checkpoint copy is the first cost. By default each node keeps its own copy of roughly 126 GiB in ~/.cache/huggingface, and start.sh rsyncs the worker's copy from the head once; later launches detect it and skip the transfer. Enabling NFS sharing removes that copy but couples the worker to the head's export, so the worker is no longer self-sufficient. Page caches are the second issue, and the README is unusually direct about it: dropping them is not done for you, it needs root, and it matters on GB10 unified memory. Skip it on both nodes and you are launching into a memory state the scripts did not create. The third constraint is the fabric. The prerequisites list ConnectX RoCE or IB between the nodes, passwordless SSH, and Docker on both. The .env.sample warns never to list an HCA port that is cabled to another cluster, and notes IB_GID_INDEX is usually 3 on ConnectX with RoCE, with 5 as the fallback if 3 does not come up. If your two Sparks are on plain Ethernet without a RoCE or IB path, this repository is the wrong tool; the parallelism it configures assumes that link. The fourth is the preflight check itself: it refuses to launch when another process holds a GPU, which is correct behaviour but will stop you if you are also running something else on either node.

How this differs from a single-node vLLM deployment

The obvious alternative is the upstream project this one is based on, getrefined/Qwen3.8-Flash-Next-NVFP4-vLLM-DGX-Spark, which the README credits. The difference is scope: the base project targets one DGX Spark, while this repository adds the second node, the rank 0 and rank 1 launch order, the worker weight distribution, and the NFS sharing option. If you have one Spark, the base project is the smaller thing to read, and the extra machinery here (rsync or NFS of a 126 GiB checkpoint, cross-node preflight, per-node page cache drops) buys you nothing. The other alternative is not running this stack at all: a hosted endpoint for the same model removes the fabric, the checkpoint copies and the patched image, at the cost of sending your traffic elsewhere. The repository's own start-tp1.sh suggests a tensor-parallel-1 path exists in the tree, but the README's prerequisites and quick start are written around two nodes, so treat single-node use as unverified by the documentation.

Licence and the cost of keeping it running

The repository is AGPL-3.0. If you run a modified version as a network service, that licence carries obligations that differ from permissive licences, and the scripts here are the kind of thing teams do modify. This is not legal advice; read the LICENSE file and decide with whoever handles licensing at your organisation. On upgrade cost, the maintenance burden is concentrated in the two patches. The PLE patch and the MXFP8 patch are extracted from the image and bind-mounted, so a new vllm/vllm-openai:qwen38-flash-next image can change the files being patched without any change to this repository. The README does not document rollback, and there are no releases listed, so pinning the image tag yourself is the only version control the documentation offers. The repository's last push was on 2026-09-16, which is recent, but a recent push is not the same as a compatibility guarantee against a moving upstream image.

Editorial conclusion

This is a fit for engineers who already own two DGX Spark nodes on a ConnectX link and want the NVFP4 checkpoint served with expert parallelism without writing their own launch plumbing. It is not a fit for single-node users, for anyone without a RoCE or IB fabric between the machines, or for teams that need a supported, versioned release: there are no releases, the project is AGPL-3.0, and it patches files inside the vLLM image at launch. Before committing, confirm that passwordless SSH works both ways, run ibstat to check the HCA name against IB_HCA, and read the KV cache lines out of docker logs vllm-fn about eleven minutes after launch to see whether the pool matches your context target.

Frequently asked questions

Official sources

1. Issues
2. License: AGPL-3.0
3. MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks on GitHub
4. Project website
5. README