jamesob/local-llm: a hardware build log for running 594B models on four RTX PRO 6000s

What this repository is, and what it is not

The README opens with a framing that tells you where the money goes: "Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence? How about $40k?" The repository is a build log and a set of serving configs, not a tool you install. The Shell language classification comes from the scripts and Docker orchestration in the tree, not from a library API. The README states plainly that nothing in it aside from the tables was written by AI, which is a small signal about how the prose should be read: as one person's notes rather than generated documentation.

The intended reader is someone assembling a multi-GPU box for local inference and willing to spend either roughly $2,000 or roughly $40,000. The README says the $20,000 tier is a TODO and that the author has no experience there, so the middle of the market is explicitly out of scope. That honesty about coverage is more useful than a padded buying guide would be, because it stops you from reading advice that was never validated.

The problem: VRAM capacity, not GPU throughput

The README's central claim is that spilling model weights into system RAM makes LLM performance unusably slow for agentic workloads. That claim is the reason the build exists in the shape it does. If you accept it, the design goal stops being "fastest possible tokens per second" and becomes "fit the model entirely in VRAM, then make the cards talk to each other cheaply."

Everything downstream follows. Four RTX PRO 6000 Blackwell Workstation cards at 96GB each give 384GB of VRAM, which the README describes as enough for something "pretty close to Claude Opus." The host system is deliberately last-generation: an EPYC Milan 7313P on an ASRock Rack ROMED8-2T, with 128GB of DDR4 ECC bought used on eBay. The README estimates that choosing a Gen4 host over a Gen5 one saves about $10,000 while keeping the VRAM budget intact. The trade being made is explicit: spend on VRAM, economize on the platform around it.

Why a PCIe switch sits between the GPUs

The unusual component is a c-payne Microchip Switchtec PM40100 Gen4 switch. The README explains the motivation in terms of tensor parallelism: during the allreduce step, GPUs need to exchange data, and without a switch that traffic goes through the PCI root complex. With the switch, the README says the cards communicate "directly" at wire speeds, which reduces inter-card latency and lowers the need for PCIe 5.0 hardware.

The sub-BOM lists the switch itself at €1,050, a SlimSAS host adapter at €140, and two SFF-8654 8i cables at roughly €30 each, for about €1,220 or roughly $1,330. The switch provides two SlimSAS 8i upstream ports, five x16 downstream slots, an auxiliary x4 SlimSAS port, and three 8-pin EPS power connectors. The README reports Gen4 line rate at 27.5 GB/s in one direction and 50.4 GB/s bidirectional, with sub-microsecond latency, measured with the included tools/measure-gpu-speed.sh script. Those figures come from the README's own result section; they are the author's measurements on his hardware, not a general guarantee.

The configuration steps that actually make it work

This is where the repository earns its keep, because the failure modes are documented alongside the fixes. Three settings are called out in the contents table as load-bearing.

First, BIOS bifurcation, link speed, and ASPM must be set correctly for the switches to behave. The README treats this as a section of its own, which suggests it is the step most likely to go wrong silently. Second, the kernel and GRUB parameters need iommu=off, or NCCL hangs. That is a concrete, checkable failure: not degraded performance, but a hang in the collective communication library that tensor parallelism depends on. Third, ACS must be disabled, which the README marks as critical for switch peer-to-peer, so that P2P traffic stays inside the switch fabric instead of being redirected through the root complex.

There is also a power constraint worth noting. The README describes running roughly $46,000 of silicon on a 110V circuit by limiting GPU power. The GPU mount itself required a day of carpentry because the author fabricated a wood enclosure for the switch and cards, and he unplugged the switch's built-in fan after finding it loud and seemingly useless.

Getting a model serving: the runners directory

The runners/ directory holds ready-to-run serving configurations. The documented example is runners/GLM-5.2-594B, described as a vLLM docker-compose setup using DCP4 and MTP5, reported at roughly 80 tokens per second at 460k context. DCP4 and MTP5 are configuration choices for that specific model and hardware combination; the README does not explain them in the excerpt available, so treat them as things to look up in the runner files themselves rather than as portable defaults.

The second runner is runners/stt, a speech-to-text configuration built around cohere-transcribe, which the README says assumes about 11GB of VRAM on an Nvidia GPU. The author notes he previously used whisper-large-v3 and finds the newer model useful. He also makes a privacy argument for local STT that is worth taking at face value: he feels comfortable using it, unlike a hosted equivalent. That is a personal judgement, not a benchmark, and the README presents it as such.

Where this repository stops being useful

The gaps are as informative as the content. The $20,000 tier is an empty TODO with three bullet points (two RTX 6000 Pro cards, four networked DGX Sparks, Apple hardware) and an explicit statement that the author has no experience with that regime. If your budget lands there, this repository will not help you.

The model table has exactly one row, dated 2026-07-01, pointing at GLM-5.2-Int8Mix-NVFP4-REAP-594B. A single recommendation with a single date is a snapshot, not a maintained compatibility matrix, and there is no indication in the material of how often it is refreshed. There are no releases retrieved for the repository, so there is no versioned artifact to pin.

The license is listed as unknown in the repository metadata. That matters more here than it would for a code library, because the material is largely prose, tables, and configuration. If you intend to reuse the BOM tables or the runner configs in your own documentation or product, the absence of a license means you cannot assume permission. That is a factual observation about the metadata, not legal advice; if reuse matters to you, ask the author directly.

Finally, the guidance is specific to one person's hardware. The iommu=off and ACS disable steps are tied to a platform with a PCIe switch in the path. On a machine without that switch, disabling ACS may be unnecessary, and the README does not discuss that case.

The alternative: four networked DGX Sparks

The README names its own alternative rather than leaving you to find one. Instead of four RTX PRO 6000 cards in one chassis, you could build a linked cluster of four DGX Spark units for a total of 512GB of VRAM, and the README links a video describing that approach.

The difference in approach is architectural, not just budgetary. The single-chassis design puts all VRAM behind one PCIe switch fabric with peer-to-peer links and sub-microsecond latency between cards, which is what makes tensor parallelism across four GPUs practical at the speeds quoted. A four-node cluster spreads memory across separate machines connected by a network, so the interconnect is no longer a switch fabric but whatever links the nodes. The README frames the cluster as a "slow, big brain" that could drive a smaller model like Qwen3.7-27b for routine work, which concedes the latency difference directly. You get more total memory and a different failure profile; you give up the tight coupling that a single switch domain provides.

Maintenance cost and who this is for

The maintenance burden here is hardware-shaped. BIOS settings, GRUB parameters, and ACS state are things that can be reset by a firmware update or a CMOS clear, and the README documents them as prerequisites rather than one-time setup, which implies they need to be re-verified after any platform change. Model runners age faster than the hardware: the single dated model entry will drift out of date as new quantizations appear, and the vLLM image referenced in the docker-compose file will need periodic attention.

Adopt this repository if you are speccing a multi-GPU inference box and want a concrete, priced reference for the switch fabric, the host platform, and the kernel-level settings that a naive build would miss. Read the runners/ directory directly rather than relying on the README summary, since the README does not explain every key it uses. Do not adopt it if you need a supported library, a licensed artifact, or guidance in the $20,000 range. The first thing to verify before buying anything is whether your motherboard's BIOS exposes the bifurcation and ACS controls the README depends on, because without them the switch is an expensive piece of metal.

Editorial conclusion

Adopt this repository as a reference if you are planning a multi-GPU workstation and want to see one fully specified build, including the PCIe switch sub-BOM, the BIOS bifurcation and ASPM settings, the iommu=off GRUB parameter, and the ACS disable step that the README ties to switch peer-to-peer traffic. Do not adopt it as a software dependency: there is no package, no installable module, and no test suite, and the license field is empty in the repository metadata. Before copying the runner configs, verify the exact vLLM image tag and model revision against your own VRAM total, because the documented GLM-5.2-594B configuration assumes 384GB across four cards and the README gives no fallback for smaller hosts.

Official sources

1. Issues
2. jamesob/local-llm on GitHub
3. README