rtx6kpro: a field wiki for running frontier LLMs on RTX PRO 6000 Blackwell PCIe cards
RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink
At a glance
- What is it?
- The repository is documentation, not software: runbooks, benchmark tables and KLD notes for serving large models on SM120 PCIe GPUs without NVLink. Its value depends on whether you are reproducing one of its qualified configurations or looking for a supported inference stack.
- Who is it for?
- Adopt rtx6kpro if you already own RTX PRO 6000 Blackwell or SM120 PCIe hardware and need a starting point for a specific model family, because the hub pages carry exact launch arguments, Docker image references and KLD methodology that are otherwise scattered across issue threads. Do not adopt it as a dependency or expect it to serve a model on its own; there is no package, no install target and no retrieved release.
- Can I use it commercially?
- Not without permission. GitHub finds no licence file in the repository, and without a licence all rights are reserved by default: you may read the code but not reuse it. Check the README, or ask the authors, before using it.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem is not inference, it is the missing manual for PCIe-only Blackwell serving
Running a 397B-class model on four PCIe cards without NVLink is a configuration problem before it is a modelling problem. The README frames the repository as a field wiki for NVIDIA RTX PRO 6000 Blackwell and SM120 PCIe systems, and lists its scope as reproducible Docker builds, vLLM and SGLang runbooks, benchmark tables, KLD quality checks, quantization notes, DCP/MTP/DSpark/DFlash debugging, PCIe topology work and regression history. That list is the actual audience definition. If you are on NVLink-connected datacenter parts, most of the PCIe topology material and the oneshot all-reduce page are not aimed at you. If you are on consumer cards, the SM120-specific quantization paths (NVFP4, MXFP8, MXFP4 experts) will not map cleanly either. The wiki sits in the narrow band of people who bought PCIe workstation cards and now want to serve quantized frontier weights across them. The README is explicit that this is documentation rather than a runtime: it points at model hubs and versioned model pages that declare a supported or qualified status, and it tells readers to prefer the hub page for each model family unless they are reproducing a specific historical result.
How the repository is organised: hub pages, versioned pages, and a generated index
The structure is a documentation graph. At the top there is a table mapping model families to start pages: GLM-5.3-Flash, GLM-5.2, DeepSeek-V4-Flash and DSpark, Kimi, Xiaomi MiMo, Qwen3.8-Flash-Next, Qwen3.8-27B, GLM-5.1, and a legacy bucket for DeepSeek-V4-Pro, GLM-4.7, Qwen, MiniMax and older Kimi pages. Each hub then links into versioned pages with names like the GLM-5.2 Infernal Invocation r18 or the DeepSeek-V4-Flash Jovian Judgement r9, which the README describes as source-locked or source-qualified configurations. Below that sit cross-cutting topic pages: Docker images and release lines, PCIe oneshot all-reduce, NCCL tuning and empty graph-file failures, speculative decoding, NVFP4 quantization, hybrid NVFP4 assembly, B12X dense FP8 GEMM versus DeepGEMM, B12X W4A8 MX tiny decode, DSpark upstream consolidation, and I/O tuning. There is also an INDEX.md described as a complete generated catalog of every page, a GLOSSARY.md with acronym expansions and writing rules, and a newcomer onboarding page. The important structural claim is the one about older pages being preserved on purpose. That means the repository accumulates rather than prunes, and a search engine result may land you on a superseded configuration. The README's own instruction is to prefer the hub page unless reproducing a historical result.
What you actually run: Docker images, vLLM and SGLang command lines
There is no pip install and no entry point in the material provided. The executable content lives inside the model pages. The GLM-5.3-Flash entry describes a Jovian Judgement Community source-locked NVFP4 target with no-spec, MTP:3, and MXFP8 DFlash2 modes, qualified TP4/DCP1 performance, AA-LCR and KLD evaluation, optional DCP4 full-CKV prefill, and a Docker launch. The DeepSeek-V4-Flash Jovian Judgement r9 page is described as source-locked TP2 fixed-K5 text and fixed-K3 Vision serving with masked-KV and stream-lifetime corrections, measured GPU KV admission, and qualified text LMCache restoration. The shared GLM/Qwen/DS4 image page mentions a two-layer R30 image, separate text K5 and Vision K3 profiles, publisher sampling defaults and immutable LMCache transfers. Those tokens (TP4, DCP1, MTP:3, K5, K3, LMCache) are the configuration surface you are expected to copy into launch commands. The repository also has a scripts directory referenced for rebuilding the Docker image, and an Eldritch Docker page. What is not in the supplied material is any literal command block, so I cannot quote a run line verbatim. Treat the runbooks as the source of the exact flags and verify them against the page you open rather than against this review.
The quality-control layer: KLD checks and a real workload comparison
Most runbook collections stop at throughput. This one carries a fidelity thread. There is a general KLD methodology page under kld/README.md, a GLM-5.2 KLD evaluation under benchmarks, and model-specific KLD sections. KLD here is used to compare a quantized model's output distribution against a reference, which is the right instrument when you are deciding whether an NVFP4 or MXFP4 checkpoint is acceptable for your task. The README also points at a Czech tutoring quality comparison covering DeepSeek-V4-Flash, V4.1-Flash, Qwen3.8-Flash-Next and GLM-5.3-Flash per reasoning mode, with a reproducible runner. That is a narrow domain, and the authors present it as such. The honest reading is that a single tutoring workload is a sanity check on output quality, not a general quality ranking, and the README does not claim otherwise. For throughput there is a consolidated Benchmark Results page and an inference-throughput directory explicitly framed as vLLM versus SGLang. The Qwen3.8-Flash-Next entry distinguishes qualified TP1/MTP3 text serving from a TP2 recipe, and separately attributes its C1 to C16 and Sieve results, which suggests the authors are trying to keep measured and reported numbers apart.
Where this repository will not help you
Three failure modes are visible from the material alone. First, it is not software. If you want a maintained serving library with a version number and a changelog, this is the wrong artifact; the supplied metadata shows no releases retrieved, and the README describes a wiki. Second, the hardware assumption is hard. The title and description both name RTX PRO 6000 Blackwell PCIe without NVLink, and the optimization pages are about PCIe oneshot all-reduce, NCCL tuning, empty graph-file failures and PCIe bandwidth. On an NVLink system those pages are at best irrelevant. Third, the versioning is a trap for casual readers. The README says older pages are intentionally preserved and that you should prefer the hub page unless reproducing a historical result, which means the repository's own longevity works against a reader who arrives from a search result. There is also a licensing gap: the supplied metadata lists the license as unknown, so the terms under which the runbooks, scripts and Docker references can be reused are not established in the material I have. That is a question to resolve before you copy build scripts into an internal repository.
How it differs from the upstream frameworks it documents
The obvious alternative is the upstream project documentation itself. vLLM and SGLang both publish their own docs, and they cover the general case: how to launch a server, how to configure tensor parallelism, how to enable speculative decoding. What they do not carry is a record of which exact combination worked on a specific PCIe Blackwell topology, at which quantization, with which KV cache admission behaviour. That is the gap this wiki fills, and it is a real one, because the failure mode on these systems is usually a combination of flags rather than a single wrong argument. The trade-off is the inverse: upstream docs track the current release and are maintained by the people who write the code, while this wiki is a snapshot with an explicit historical archive. A second alternative is a vendor or cloud serving stack, which removes the topology problem entirely by not running on your hardware. If your goal is to serve a model rather than to operate GPUs, that is the cheaper path. The wiki only pays off when the hardware is already bought and the constraint is non-negotiable.
Maintenance cost and what to check before trusting a page
The last push recorded in the metadata is 2026-09-10, and the repository is not archived, so it is being updated. The README's own regression history and daily summaries sections imply ongoing work, and the presence of qualified versus supported status labels implies the authors distinguish configurations they have validated from ones they have merely run. That distinction is the thing to look for. When you open a model page, check whether it carries a qualified status and a version tag, and check whether the hub page for that family points to it. If the page you found is not linked from the hub, you are probably reading a preserved historical result. On licensing, the repository metadata shows no license, so redistribution of the Docker references or scripts is unresolved and I cannot advise on it; treat that as an open question for whoever owns your compliance review. The maintenance cost for you is not installing anything, it is the reading time required to find the current page and the discipline to re-check it after a model or driver update, since the runbooks encode specific CUDA and image versions.
Editorial conclusion
Adopt rtx6kpro if you already own RTX PRO 6000 Blackwell or SM120 PCIe hardware and need a starting point for a specific model family, because the hub pages carry exact launch arguments, Docker image references and KLD methodology that are otherwise scattered across issue threads. Do not adopt it as a dependency or expect it to serve a model on its own; there is no package, no install target and no retrieved release. Before following any page, verify the license status of the repository itself, since the supplied metadata lists it as unknown, and confirm that the page you are reading is the current hub rather than one of the preserved historical pages the README says are kept intentionally.
Community notes