DeepSeek v4 Flash EXL3 on one DGX Spark: a single-node serving recipe
DeepSeek v4 Flash EXL3 on one DGX Spark
At a glance
- What is it?
- MiaAI-Lab's launcher runs the 3.0 bpw EXL3 build of DeepSeek V4 Flash 0731 on one GB10 box with DSpark speculative decoding, a compressed NVFP4 KV cache and a 384k context default. It is a tuned Docker recipe for a specific machine, not a portable inference stack.
- Who is it for?
- Adopt this if you already own a DGX Spark with 128 GiB unified memory and want the 3.0 bpw EXL3 build of DeepSeek V4 Flash 0731 served locally at roughly 44 to 47 tok/s without a second node. Do not adopt it on any other GPU, on x86, or on a machine you share with other workloads, because it holds about 94% of unified memory by design.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 25 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 18, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: a 3.0 bpw EXL3 build that fits on one Spark
The official FP4 build of DeepSeek V4 Flash is documented as needing tensor parallelism across two DGX Sparks. This project exists to avoid the second box. It serves the `0xSero/deepseek-v4-flash-0731-spark` checkpoint, a 3.0 bits-per-weight EXL3 quantisation, through the `sparkinfer` kernel stack (formerly called `b12x`) at tensor-parallel 1. The audience is narrow and obvious: someone who already has one GB10 machine and does not want to buy or borrow another. Everything in the repository is arranged around that constraint. The launcher pins the KV cache to a compressed NVFP4 format and the attention and MoE path to the `B12X_MLA_SPARSE` backend, both of which exist to buy context length back from the memory the weights consumed. If you are not on a DGX Spark, none of this applies to you, and the runtime image is aarch64-only.
How the launcher, DSpark and the KV pool fit together
`start.sh` is the entry point. It writes a `compose.yml`, pulls the runtime image, downloads the weights into `./hf-hub`, and brings up the server. Two upstream kernel backports ship as read-only bind-mounts from `image-patch/sparkinfer/`, which is how the repository applies fixes without rebuilding the image. Speculative decoding runs in `MODE=dspark` with a fixed K5 draft and a K64 draft model, and CUDA-graph capture is tuned to `[6,12,24]` so concurrent decode stays on captured graphs rather than falling back to eager execution. The memory story is the interesting part. The model keeps 128-token sliding-window layers alongside full-depth global layers, and how the KV byte budget splits between them shifts with `MAX_NUM_SEQS`. The README's own explanation is that single-sequence boots route more of the budget into the global cache that defines the reported pool, which is why a 2-sequence boot reported 337,841 tokens while comparable bytes at 1 sequence reported over 402,000. Cold versus warm JIT adds a second swing of roughly 0.6 GiB, about 28k tokens, because a boot compiling fresh kernels leaves less free memory at KV-sizing time. The pool number is therefore leftover-derived, not fixed.
Quick start: install and first request
The README's quick start is two commands. The default path launches the deep-context NVFP4 configuration with a 384k `MAX_MODEL_LEN` and DSpark, and writes `compose.yml` on the way. Run it from the repository root.
./start.sh # start: deep-context NVFP4 (384k, DSpark) - writes compose.yml
./start.sh --no-wait # start without waitingFirst boot is described as intentionally slow because of the weight download. The README puts the download at roughly 107 GB and asks for about 110+ GiB of free local disk, so budget the time and the space before you start. No HuggingFace login is required; the repository and image are public, and an optional `HF_TOKEN` exists for rate limits or private repos.
The defaults were changed on 2026-08-21. The launcher now serves `KV_RECORD=stock432` with native 432-byte records, `GPU_MEMORY_UTILIZATION=0.94`, `MAX_MODEL_LEN=384000` and `MAX_NUM_SEQS=1`. Those values are what produce the 439,622-token KV pool quoted in the README, and they are also why the machine is effectively dedicated to this server while it runs.
Once the server is up, the README describes a `/health` endpoint returning 200 and KV usage returning to 0% between requests. A first real use is a long-context recall check rather than a chat prompt, because that is what the project was tuned for. The stress test method in the README is reproducible: one large user prompt of random-word filler generated so prefix caching cannot deduplicate it, a passphrase planted near the start, a recall question at the end, `temperature 0`, thinking disabled through `chat_template_kwargs`, and `max_completion_tokens 128`. Expect prefill to decay with depth. The README reports roughly 1,024 tok/s at the start of a request and 350 to 614 tok/s past 300k accumulated tokens, which is why a full 384k prefill takes around ten minutes end to end.
The 439,622-token pool is a boot-time number, not a spec
This is the part a buyer should read twice. The README lists observed pool sizes of 337,841, 402,334, 430,909, 440,461 and 439,622 across different boots on the same host. The 439,622 figure is a cold boot at 384k with one sequence, not a guarantee. The README states that the worst boot observed still cleared `MAX_MODEL_LEN` with at least 1.14x headroom, and that a genuinely bad boot trips the boot-time KV check and stops cleanly under `restart: on-failure:1`. That is a reasonable failure mode, but it means your usable context is decided at startup, and a warm JIT cache can matter by roughly 28k tokens. If your workload assumes a fixed context ceiling, measure it on your own machine rather than copying the table. The decode figure of 44 to 47 tok/s is likewise reported for the structured workload described, at 384k context, not a general throughput claim.
EarlyOOM and the 94% memory bet
`GPU_MEMORY_UTILIZATION=0.94` means the server intentionally holds about 94% of the 128 GiB unified memory. The README is direct about the consequence: a user-space OOM killer cannot distinguish a healthy server from a leak and may kill it mid-serve, so it instructs you to disable EarlyOOM if present.
sudo systemctl disable --now earlyoomThat single line is the clearest signal of what this project is. It is not a stack you co-locate with other services on the same box. It assumes the DGX Spark is doing this and little else while the server runs. If you need the machine for other work, this is the wrong tool, and the optional LAN sharing mode does not change that: it reuses one ~107 GB weight copy across machines over an SSHFS share, which needs `sshfs`, `fuse3` and `user_allow_other` in `/etc/fuse.conf`, but the memory footprint on the serving node is unchanged.
What the README does not cover
There is no rollback path documented. The defaults changed on 2026-08-21 and the README refers to an internal postmortem about NVFP4 dual-cache prefill bugs that is explicitly kept local and not in the repository, so the reasoning behind the old either/or behaviour is not available to you. The pre-fix kernel is described as producing NaN on any prompt of 7 tokens or more; the fixed path is what the 320k and 370k runs exercised. That is a strong argument for pinning to the current commit rather than tracking the branch. There is also no release history in the repository to pin against, and no documented upgrade procedure. The repository ships `download.sh`, `start.sh` and `stop.sh` at the top level, plus `files/` and `image-patch/`, and that is the whole surface. A team that needs a supported upgrade channel will not find one here.
How this differs from a general serving stack
The obvious alternative is vLLM, which most people reach for when serving a quantised model on a single GPU. The difference is not quality, it is scope. vLLM targets many architectures, many quantisation formats and multi-GPU topologies, and you configure it yourself. This project is the opposite: a fixed recipe for one checkpoint on one board, with the kernel backports, CUDA-graph capture list, KV record format and memory utilisation already chosen. The official FP4 build of DeepSeek V4 Flash is the other reference point, and the README's stated distinction is that it needs tensor parallelism across two Sparks where this runs at TP1. If you are not on a DGX Spark, vLLM is the more sensible starting point because this launcher's assumptions do not transfer. If you are on a Spark and want the second node out of the picture, the trade you accept is that every knob is already set and the project's own documentation is the only support channel.
Licence and maintenance
The repository is MIT licensed, which covers the launcher scripts and the bind-mounted patches. It does not automatically cover the model weights or the runtime image, which are separate artifacts with their own terms, and the README does not state what those are. Treat the MIT grant as applying to the code in this repository and check the weight and image licences separately before any commercial deployment. The last push was on 2026-08-24, and the repository is not archived. There are no releases retrieved, so there is no tagged version to upgrade between; the practical upgrade path is pulling a newer commit and re-running `start.sh`, which rewrites `compose.yml`. Because the defaults changed once already, on 2026-08-21, expect that a future pull can silently change your context or memory configuration. Record the commit you validated.
Editorial conclusion
Adopt this if you already own a DGX Spark with 128 GiB unified memory and want the 3.0 bpw EXL3 build of DeepSeek V4 Flash 0731 served locally at roughly 44 to 47 tok/s without a second node. Do not adopt it on any other GPU, on x86, or on a machine you share with other workloads, because it holds about 94% of unified memory by design. Before you commit the ~107 GB download, verify three things: that EarlyOOM is disabled, that the aarch64 runtime image matches your DGX OS, and that the boot-time KV check passes with the pool size your own boot reports rather than the 439,622-token figure from the README.
Frequently asked questions
Can I run DeepSeek V4 on DGX Spark?
This repository is a single-node launcher that does exactly that, serving the 3.0 bpw EXL3 build of DeepSeek V4 Flash 0731 on one DGX Spark at tensor-parallel 1. It requires GB10 hardware with at least 128 GiB unified memory, Linux aarch64, and the NVIDIA Container Toolkit for GPU passthrough to Docker.
What operating system does the DGX Spark run for this launcher?
The README specifies Linux aarch64, DGX OS, and notes that the runtime image is aarch64-only. The host also needs Docker Engine plus Compose v2, curl, and roughly 110+ GiB of free local disk.
Is DGX Spark using unified memory, and does that matter here?
Yes. The launcher assumes 128 GiB of unified memory and sets GPU_MEMORY_UTILIZATION=0.94, so the server intentionally holds about 94% of it. That is why the README tells you to disable EarlyOOM on the host: a user-space OOM killer may terminate the server mid-serve.
What are people using the DGX Spark for with this project?
The README frames it around long-context serving: the default configuration targets a 384k MAX_MODEL_LEN with a 439,622-token KV pool, and the stress test pushes 320k and 370k token prompts through it. The measured decode figure is 44 to 47 tok/s at 384k context.
Community notes