# heterogeneous-gpu-pd-lab: splitting prefill and decode across two very different GPUs

> This repository is an experiment record for running a hybrid MoE model across a dense accelerator and a large-memory host, with the prefill and decode work divided between them over RDMA. The measurements are raw by design, and the README is explicit about where the comparisons stop holding.

**Soulmate-Halo/heterogeneous-gpu-pd-lab** — 使用稠密算力卡对稀疏大显存主机进行加速，主机包括 AI Max+ 395、M3 512G、DGX Spark 等。

- Repository: https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab
- Stars: 917 · Forks: 142
- Language: C++
- License: not declared
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/soulmate-halo-heterogeneous-gpu-pd-lab

## A lab notebook, not a framework

The repository name says lab and the README behaves like one. Its subject is heterogeneous acceleration for large-memory hosts: the project description states that it uses dense compute cards to accelerate sparse large-memory hosts, naming AI Max+ 395, M3 512G and DGX Spark among the host machines. The repository topics list llama-cpp, pipeline-parallelism, prefill-decode, strix-halo and rtx-3060, which tells you both the workload family and the hardware mix.

What you get is a set of dated experiment write-ups, each with a table of measurements, a deployment diagram and a bundle directory. The top level is a row of those bundles: qwen38-flash-spark-3080, qwen38-flash-395-3080, ds41-flash-v10, ds41-flash-v8, plus data/, results/, scripts named make_base_architecture.py and make_dense_region_structure.py, a CHANGELOG in two languages, and a VERSION file.

There is no library API here to call and no package to install into your inference stack. The value for a reader is the mechanism and the layout. The mechanism is a division of labour between two machines that are badly matched to each other, and the layout shows how that division is carried over a network. The rest is the numbers, which the README itself treats with suspicion.

The project description is written in Chinese and the README links a Chinese version alongside the English one, so assume some of the detail lives in the translated files rather than in the page you read.

## How the split is carried: RDMA expert exchange, not an HTTP handoff

The central technical claim in the README is about how the two machines talk to each other. For the Qwen3.8 Flash-Next V2 setup with an RTX 3080 and a DGX Spark, the diagram describes the division: the Spark handles Prefill alone, and then Spark and the 3080 cooperate on Decode, with the 3080 taking hot experts and main compute while the Spark handles cold expert compute.

The diagram as published is:

```mermaid
flowchart LR
    A["API request"] --> P["DGX Spark / GB10<br/>Solo Prefill"]
    P --> D["Decode work distribution"]
    D --> G["RTX 3080 20GB<br/>Hot experts and main compute"]
    D --> S["DGX Spark / GB10<br/>Cold expert compute"]
    G <-->|"RDMA expert requests and results"| S
    G --> O["Output tokens<br/>Faster joint Decode"]
```

The interesting line is the bidirectional RDMA link between the two expert hosts, and the README is careful about what that link is not. The frozen HTTP router selects a backend per request, has no cross-host KV export or import interface, and cannot hand one request from Spark Prefill to 3080 Decode. Joint Decode instead relies on RDMA compute cooperation between the hot experts on the 3080 and the cold experts on Spark, rather than polling two independent HTTP services.

That is a meaningful constraint for anyone planning to reuse this design. If your serving stack expects to move a request, including its KV cache, from one inference server to another, this layout does not offer that. The work is split at the level of expert computation, not at the level of request routing.

## What the Spark and 3080 decode numbers do not establish

The V2 table is the most quotable part of the README, and the paragraph under it is the most important. The reported figures are an aggregate Decode of 242.37 tok/s against a 107.60 tok/s Spark solo baseline, and a single stream Decode of 62.00 tok/s against 22.30 tok/s solo. The README itself converts these to roughly 2.25x aggregate and 2.78x single-stream decode.

Then it takes that back, in part. The V2 Decode results and the solo controls came from the experimenter on 2026-10-06, and raw benchmark logs, concurrency details and workload details were not included in this revision. The three public Prefill runs use 3600 prompt tokens while the solo Prefill control uses 2047, so the README states they do not establish a strict Prefill speedup even though the Prefill mean of 1118.43 tok/s sits right next to the 1117.93 tok/s solo figure.

This is the correct way to read the table. The decode ratios are what the author chose to report, but they are reported without the raw logs that would let you check them, and the prefill column should not be read as a win. An engineer budgeting a deployment should treat 242.37 tok/s as a number attached to a specific configuration on a specific date, not as a property of the architecture.

## AI Max 395 with an RTX 3080: the gain is in prefill, decode stays flat

The second setup pairs a discrete RTX 3080 20GB over OCuLink with an AI Max 395, described as an 8060S iGPU plus a Zen5 CPU with AVX-512. The 3080 computes hot experts and the dense main path, while the AI Max 395 computes cold experts. The README states that the 3080 accelerates prefill about 3x.

The table gives exact figures. Prefill is 843.70 tok/s against an AI Max 395 solo baseline of 273.04 tok/s, and the README expands the 843.70 figure as the measured 16K tier peak, with the 3080 holding 55 percent of layers and ub 1024, alongside pp1K 612.40, pp6K 797.00 and pp16K 843.70. Decode is 40.55 tok/s against a 41.33 tok/s solo baseline.

The decode comparison is the interesting one, because it goes the other way. Decode runs mostly on the AI Max 395 in both deployments, so it stays flat, and the README attributes the decode range of 28 to 56 tok/s to the expert-cache hit rate rising from 47 percent to 83 percent. The claimed prefill gain is about 3.1x, from 273.04 to 843.70 tok/s.

So the pattern across the two setups is consistent and worth stating plainly: the accelerator helps the compute-heavy prefill phase, while decode on a large-memory host with a big CPU and iGPU is already competitive enough that the paired GPU adds little there. The model is a 125B Qwen3.8-Flash-Next hybrid MoE in NVFP4, served by the Strata NVFP4 engine on a single host named mx7.

## DS4.1 Flash V10: one 6000D in front of four Spark nodes

The third and most recent experiment, dated 2026-10-14, scales the idea. One RTX 6000D runs all Prefill as a 21 of 40 layer prefill-only slice at TP1, while four DGX Spark nodes run Decode through SGLang at TP4. KV crosses hosts over NIXL/RoCE, and the Engram tables are served remotely by 8 shard servers running on the Spark nodes. The README notes that this layout applies to any DS4.1 deployment where 6000D acceleration is available.

The comparison table lists three setups, and the README immediately warns against reading ratios across them. V10 with one 6000D and four Spark nodes reports Prefill of 10082, 10308, 10372 and 10340 tok/s across concurrency levels C1, C2, C4 and C8 at 100K input, and an aggregate Decode of 214.3 tok/s at 100K context under C8. V8 with two 6000D and four Spark nodes reports 16326.03, 16297.20 and 16297.81 tok/s of Prefill, but at 32K input and concurrency C16 to C32. The four-node Spark TP4 baseline reports 4138.75 tok/s single stream.

Because input lengths, concurrency levels and workloads differ across the three rows, the README says to treat them as order-of-magnitude references only, not as a basis for cross-setup speedup ratios. The same paragraph also notes that the single 6000D sustains a marginal Prefill rate of 10195 tok/s, which is above the per-card share of the official dual-card TP2 deployment at 12996 tok/s total.

A separate line gives the honest end-to-end number: 100K input, 500 out, C8, end-to-end aggregate of 44.72 tok/s. That is the figure to use when you size a real service.

## Getting the containers: images that exclude the heavy parts

The deployment path runs through GHCR images rather than a package manager. For the Spark and 3080 setup the image is ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/qwen38-flash-spark-3080:latest, and the README says it includes CUDA 13 build tools, RDMA dependencies, pinned TVM-FFI, public sources, role commands and startup diagnostics. The first native build is cached on each host.

For DS4.1 Flash V10 the image is ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/ds41-flash-v10:latest, which ships deployment configs, launch templates and verification scripts. What it does not ship matters more for planning: the 130 GB vendor role-image package and the 232 GB weights are excluded.

The README is explicit that model assets, GPU drivers and RDMA must be prepared first, and points at the Docker deployment section of the relevant guide for the details. So the practical workflow is: get the image, prepare drivers and RDMA on both hosts, fetch the assets the image leaves out, then use the role commands and verification scripts inside the image. The bundle directories in the repository, such as qwen38-flash-spark-3080/, and the evidence files referenced as v2-metrics.json and performance-v2.json are where the configuration and the numbers live.

## Why the project publishes untuned numbers on purpose

The README carries a methodology note that explains the philosophy and also the limits. It says all experiment results in the repository are untuned raw numbers, that only the base stack's built-in optimizations are used, and that no additional tuning is applied. Comparing tuned figures from other open source projects against these raw numbers is meaningless, because those techniques apply equally to heterogeneous setups and are left off only to keep the measurements clean.

That is a defensible choice and it changes how you should read the whole repository. There is no tuned number to compare against, no tuning script, and no claim that this configuration is the fastest possible. What you can compare is mechanism: how the roles are assigned, which side handles prefill, which side handles decode, how the experts are requested over RDMA, and how KV crosses hosts for the four-node case.

There is also a licensing wrinkle worth flagging. No licence identifier is listed for the repository, and there is no LICENSE file in the top level listing. The README does not state licence terms either. For an experiment record that is often fine, since the useful output is the write-up, but if you plan to reuse deployment configs, scripts or the container images inside a commercial product, that gap has to be closed by asking the maintainer rather than by reading a file.

## Conclusion

Take this repository as a set of documented experiments rather than a product. It is worth studying if you run hybrid MoE models on a machine with both a discrete accelerator and a large-memory host, because it shows the role split, the container images and the deployment layout in detail. It is not a library to embed and it is not a benchmark you can cite, since the README labels the figures untuned raw numbers with missing logs, differing workloads and no cross-setup ratios. Before you plan a deployment around it, pull the deployment guide for your own machine pair, build the matching GHCR image, and confirm you can supply the model assets, GPU drivers and RDMA setup that the images deliberately exclude.

## FAQ

### What does heterogeneous-gpu-pd-lab actually provide?

It is a set of dated experiment write-ups with deployment diagrams, bundles of configuration files, GHCR container images and raw measurement tables for hybrid MoE models served across a dense GPU and a large-memory host. It is not a library or an API that you call from your own code.

### How does the repository split prefill and decode across the two machines?

In the RTX 3080 plus DGX Spark setup, the Spark handles prefill alone and both machines cooperate on decode, with the 3080 taking hot experts and main compute while the Spark takes cold expert compute. The two sides exchange expert requests and results over RDMA rather than through two independent HTTP services.

### Can I use the numbers in the README as benchmarks?

No. The README states that all results are untuned raw numbers, that raw benchmark logs and workload details were not included for the V2 decode figures, and that the three comparison rows use different input lengths and concurrency levels. It asks that they be treated as order-of-magnitude references only.

### What do the container images include, and what must I provide?

The Spark and 3080 image includes CUDA 13 build tools, RDMA dependencies, pinned TVM-FFI, role commands and startup diagnostics, while the DS4.1 image ships configs, launch templates and verification scripts but excludes the 130 GB vendor role-image package and the 232 GB weights. Model assets, GPU drivers and RDMA must be prepared before running either.

## Sources

- [Issues](https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab/issues)
- [README](https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab/blob/main/README.md)
- [Releases](https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab/releases)
- [Soulmate-Halo/heterogeneous-gpu-pd-lab on GitHub](https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/soulmate-halo-heterogeneous-gpu-pd-lab