Lucebox: a speculative inference server for consumer GPUs and mixed AMD/NVIDIA machines
LLM speculative inference server for heterogeneous hardware & consumer GPUs
At a glance
- What is it?
- Lucebox is a C++17 inference server that pairs per-model drafters with hand-written kernels for specific GPUs. The README documents large speedups on cards like the RTX 3090 and R9700, but every number is tied to one model and one hardware target.
- Who is it for?
- Adopt Lucebox if you already own one of the cards in its tested-machines table and are willing to run the exact model and drafter pair the project measured, because the speedups come from kernels tuned per target rather than from a general runtime. Do not adopt it if you need broad model coverage, a stable release cadence, or a build that does not depend on matching a gfx or sm architecture by hand.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Lucebox targets: fast local inference on hardware you already own
Most local inference stacks optimise for one vendor and one generation of accelerators. Lucebox takes the opposite position. Its README describes an engine built for heterogeneous machines and consumer GPUs, with custom kernels, speculative prefill and speculative decoding tuned for each model and hardware target. The tested-machines table spans RDNA4 (gfx1201), RDNA3.5 (gfx1151), RDNA3 (gfx1100), Ampere (sm_86) and Blackwell (sm_120), covering an R9700, a Ryzen AI MAX+ 395, a 7900 XTX, an RTX 3090 and an RTX 5090. The audience is therefore narrow and specific: people who own one of those cards or APUs, want to serve a model locally, and are willing to accept a per-target build in exchange for throughput. The README's heterogeneous execution row describes DeepSeek V4 running across an R9700 plus Strix Halo at 86 tok/s decode and 788 tok/s prefill at 2K, which is the case a single-vendor runtime cannot address at all.
Speculative decoding with per-model drafters, not a generic runtime
The mechanism visible in the README is a target model plus a separate drafter checkpoint, with kernels specialised for the pair. Qwen 3.8 27B UD-IQ4_XS on an R9700 uses a DFlash2 source checkpoint converted to Q8_0; Laguna XS 2.1 33B uses either PFlash/KVFlash with a Qwen3 0.6B Q8_0 drafter for prefill or a DFlash Q4 drafter for decode. Gemma 4 26B-A4B and Gemma 4 31B IT each pair with their own DFlash Q8_0 drafter. DeepSeek V4 Flash uses DSpark with a Q4RMFP4 denseF16 drafter. This is the trade-off at the centre of the project: the speedups are real in the measured setups, but they are bought by shipping a specific checkpoint per model. The README states 6.4x decode versus Lucebox's own autoregressive path and 3.8x versus llama.cpp with the same drafter for Qwen 3.8 27B on an R9700. Those are the project's own measurements in its own table, and the comparison baseline for the second figure is llama.cpp using the identical drafter, which is the fairer of the two.
Prefill is treated as a first-class cost, not an afterthought
The PFlash and KVFlash rows address long-context prefill, which is where many local setups stall. The README reports Laguna XS 2.1 33B at 256K context on an RTX 3090 going from 411 s to 67.3 s, a 6.1x prefill improvement, and 152.3 tok/s with an 8K pool. Luce Spark is listed separately for the same model on the same card at roughly 100 tok/s inside 14.6 GiB. A megakernel entry covers Qwen 3.5 0.8B on an RTX 3090 at 413 tok/s and 1.87 tok/J, with results in optimizations/megakernel/RESULTS.md. Paged attention and continuous batching are also present, with the README citing 300.9 tok/s total at 5 clients for Qwen 3.8 27B plus DFlash2 on an R9700. The pattern across all of these is that Lucebox attacks the two phases separately and publishes a number for each, rather than reporting a single headline throughput figure that hides which phase improved.
Getting it running: CMake, gfx targets and the server quick start
The build requirements are stated in the README badges: CUDA 12+, HIP 7+, C++17. The tested-machines table adds runtime detail, listing ROCm 7.2 for the R9700 and Strix Halo rows and ROCm 6+ for RDNA3. Two build constraints are called out explicitly. NVIDIA architectures are selected by CMake. For HIP, the README says builds should target the device's exact gfx architecture, and the table supplies the values: gfx1201 for the R9700, gfx1151 for Strix Halo, gfx1100 for the 7900 XT and XTX. Getting that string wrong is the first thing that will break a HIP build, and the README does not present a fallback for a card whose gfx target is absent from the table. The README points to a run-the-server section for the Qwen 3.8 R9700 quick start and to server/docs/RECOMMENDED_SETUPS.md for the DeepSeek V4 Strix Halo profile, and to server/docs/DS4.md for the dual AMD 7900 XT plus Strix Halo configuration. The repository layout implied by those paths is a server directory holding docs and setup profiles, plus an optimizations directory holding per-technique READMEs and results files. The README does not list a single copy-paste command line in the material supplied here, so the exact invocation has to come from those linked documents.
What the model table does not tell you
Every speedup in the README is conditional. The Qwen 3.8 27B figure depends on converting the DFlash2 source checkpoint to Q8_0 yourself, since the table links the source weights rather than a ready quant. The DeepSeek V4 Flash row requires a ROCmFPX MIX Strix GGUF from the Lucebox Hugging Face organisation, which is a repackaged variant rather than an upstream release. The Ling 3.0 Flash 124B-A5.1B row lists 34.6 tok/s median in autoregressive mode on a DGX Spark, with no drafter, so it is not a speculative result at all. The heterogeneous execution row gives 86 tok/s decode for DeepSeek V4 across two devices, which is well below the single-device speculative numbers elsewhere in the table and reflects the cost of splitting a model across an R9700 and a Strix Halo. The README also states there are no recent releases retrieved, so the project is tracking main rather than a tagged version. Anyone expecting to pip install a wheel and point it at an arbitrary GGUF is looking at the wrong project. This is a build-from-source engine with a curated model list.
Where llama.cpp fits, and where it does not
The README itself makes the comparison that matters: 3.8x versus llama.cpp with the same drafter for Qwen 3.8 27B on an R9700. That framing is more useful than a bare speedup figure because it isolates the engine from the drafter. llama.cpp runs a far wider range of quantisations and architectures and does not require matching a gfx target or converting a drafter checkpoint by hand. Lucebox trades that breadth for kernels written against specific models and specific silicon, and the README's own numbers show the trade paying off for the pairs it has tuned. The practical difference is in what happens when your model is not in the table. With llama.cpp you run it and accept whatever speculative decoding support exists. With Lucebox you either find a drafter under the Lucebox Hugging Face organisation or you are running the autoregressive path, and the README gives no indication of what the autoregressive throughput looks like for unlisted models.
Licence, maintenance and what to check before you build
Lucebox is Apache-2.0, which permits commercial use and modification, and the README links a LICENSE file at the repository root. That covers the engine code. It does not automatically cover the model weights and drafters, which carry their own terms on Hugging Face, and the README links to third-party repositories such as unsloth, bartowski, poolside and bloomer010 alongside the Lucebox-hosted ones. Check each checkpoint's licence separately; the Apache-2.0 badge on the repository says nothing about them. On maintenance, the last push recorded is 2026-09-10 and no releases were retrieved, so there is no version number to pin and no changelog to read. Upgrading means tracking main, and a kernel tuned for gfx1201 or sm_86 can be affected by a ROCm or CUDA toolkit change independently of the repository. The blog links in the README (for example the Qwen 3.8 R9700, DeepSeek V4 Flash and Laguna XS 2.1 posts) are the closest thing to release notes available in this material, and they are per-result write-ups rather than upgrade guidance. This article is based on the README and repository metadata only; the project has not been built or run here, and no throughput figure above has been independently reproduced.
Editorial conclusion
Adopt Lucebox if you already own one of the cards in its tested-machines table and are willing to run the exact model and drafter pair the project measured, because the speedups come from kernels tuned per target rather than from a general runtime. Do not adopt it if you need broad model coverage, a stable release cadence, or a build that does not depend on matching a gfx or sm architecture by hand. Before committing, verify that a drafter exists for your model on Hugging Face under the Lucebox organisation, that your ROCm or CUDA version meets the badge minimum, and that your GPU's exact gfx target is listed in the README table.
Community notes