halogen-flash-server: Qwen3.8-Flash-Next on Strix Halo, at 5.53 bpw

One GPU, one model family, no fallback path

halogen-flash-server exists to serve Qwen3.8-Flash-Next on AMD's Strix Halo silicon, target gfx1151. The README is blunt about the method: every kernel is written for this one GPU and this one model family, with no general-purpose runtime, no portability layer and no fallback path. That is the whole design argument. Generality costs something, and this project declines to pay it.

The intended user is someone who owns a Strix Halo machine and wants to run this particular mixture-of-experts model locally, over an OpenAI-compatible HTTP endpoint, at a precision near 5.5 bits per weight. It is not a framework for experimenting with architectures, and it is not a way to run a different Qwen checkpoint. The README frames both the capability and the restriction as the same decision.

What makes the trade legible is the measurement table. On a 32K prompt with a 256-token answer, the project's own row reads 5.53 bpw, 23.0 s prefill, 6.1 s decode, 29.1 s total. The three comparison rows are, in the README's words, their own published figures on their own machines: EngramHalo.cpp at 3.71 bpw and 118.0 s, ROCmFP4 at 5.51 bpw and 117.9 s, CIRU-IU4 at 5.96 bpw and 154.7 s. The project's claim is roughly 4x faster end to end than the best of them. Because prefill dominates wall clock on any prompt with real context, that is where the claim is won.

Where the speed claim is strong and where it is soft

The README separates its two columns by confidence, which is unusual and worth reading closely. Prefill is the strong one: 23.0 s at 32,768 tokens, or 1,424 tok/s. Decode is called "the soft one" in the text. The served speculative rate is 41.7 tok/s, and the README concedes that for two of the three competitors it cannot tell whether speculative decoding was on, while EngramHalo's 14.3 s is explicitly its non-speculative number.

Rather than leave it there, the README does the arithmetic in public: hand every competitor its best plausible speculative decode and the totals still land around 110 s against 31.1 s. That is a fair way to argue, because it concedes the uncertain column and shows the conclusion survives. The prefill column carries the claim, and on prefill there is no such ambiguity.

The precision figure is also derived rather than asserted. Bits per weight is measured from the checkpoint's own tensor table instead of quoted from a format name: 5.53 bpw across all 179.55B parameters, or 4.55 bpw across the trunk and experts with the FP8 n-gram lookup table set aside. docs/QUANT.md is said to give the breakdown by tensor family and the derivation, so the number can be checked with arithmetic. That is a materially different posture from naming a quant format and letting the name do the work.

Installing it and serving a first request

The README gives a single container command as the whole quickstart. It fetches the weights on first start (118 GiB, and the transfer resumes if interrupted) and serves an OpenAI-compatible endpoint on port 8731. The models volume is mounted read-write here because it is the download target.

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.11.1

If you would rather fetch the weights yourself, the README gives the two-step form. The second command mounts the volume read-only, and the weights repo carries the tokenizer, so one volume is all either form needs.

hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir ~/halogen-models

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -v ~/halogen-models:/models:ro \
  ghcr.io/peonist-ai/halogen-flash-server:0.11.1

On Docker rather than Podman, the README says to replace --group-add keep-groups with --group-add video --group-add render, because keep-groups is a Podman extension. The shipped docker-compose.yml repeats that note for docker compose.

The two-container topology is the other deployment shape, and the compose file explains why it exists: the engine needs the GPU and re-reads the weights, while the front-end needs a tokenizer and starts in a second, so a front-end change redeploys without a model reload. The file's usage lines are these.

HALOGEN_MODELS=~/halogen-models podman-compose up -d
podman-compose restart api          # cheap: no model reload
podman-compose logs -f engine       # watch the load, it takes minutes

The engine port is deliberately not published, because the compose file states the token protocol has no auth. Only the API is reachable.

Two containers, one tag, and the failure that follows

The compose file's most consequential rule is that both services must run the same image tag. The API renders the prompt and the engine runs it, and the file states that what one release can do the other may not know how to ask for. It cites public issue #26 as the concrete case: an api at 0.4.4 in front of an engine at 0.5.6, where the front-end predated images and therefore could not request them.

That is a real operational constraint rather than a packaging detail. If you split the roles to get cheap front-end redeploys, you inherit a version-coupling obligation that the single-container form does not impose. The compose file says as much: if you do not need the split, the single container is simpler.

The download behaviour has a second edge. A start with HALOGEN_DOWNLOAD set and a writable volume re-fetches the 2.4 GiB quality sidecar when the one on disk predates the image, because 0.6.0 changed that file. The 115 GiB checkpoint is never re-fetched. With HALOGEN_DOWNLOAD unset the container opens no outbound connections at all, and the README says it reports at startup if the sidecar is the older one. So the read-write mount is not a permanent requirement, it is a property of the download path.

Speculation that is not supposed to change the answer

The identity guarantee is the most interesting claim in the README. At temperature 0, output is byte-identical to serial greedy decode. Speculation is presented as a pure speed optimization, verified on every release, rather than a quality trade. Since 0.6.0 there are two draft sources, the model's own draft head and the request's own text (prompt lookup), and the README says the guarantee covers both.

That matters because speculative decoding is normally where you accept a small quality risk in exchange for throughput. Here the risk is claimed to be zero at greedy sampling, and the claim is stated in a form that can be falsified by comparison against a serial decode. The README does not say what happens at temperature above 0, and it should not be read as making the same promise there.

Since 0.7.0 the engine also opens a llama.cpp GGUF of this model directly. Point it at a file you already have, unsloth's UD-IQ4_XS for example, and it runs on these kernels with the same speculation and the same identity guarantee. The README's framing is same file, faster runtime, no conversion step. Whether that holds for every GGUF of this model is not something the README settles; it points to a section on which files qualify.

Precision as a dial, and the empty answer that is not a bug

The README treats precision as something you trade deliberately, with a dedicated section on what you get and how to trade it, and a docs/QUANT.md breakdown by tensor family. The headline 5.53 bpw is not the lowest number in the comparison table; EngramHalo.cpp runs at 3.71 bpw, two thirds of the precision, and is the slowest of the three alternatives by total time. The one runtime carrying more bits than halogen-flash, CIRU-IU4 at 5.96 bpw, is the slowest overall. So the project's position is not that it wins on bits, it is that it wins on time at comparable or higher precision.

Configuration is where the operational complexity lives. The README documents sampling, images, token budgets, Codex and the Responses API, use from an agent harness, cache modes, context and memory with one KV pool serving several conversations, an opt-in 1M context configuration, an attention budget configuration, and a preview called composable context. That is a wide surface for a project whose selling point is doing one thing.

The token budget behaviour is the one to internalise first. The README's own heading pairs token budgets with an explanation of why an empty answer means you ran out, which is a failure mode that reads like a model problem and is actually a configuration problem. If you set a budget too low, the symptom is silence, not an error. Nothing in the README suggests the server guesses a larger budget for you.

What it is not, and what the licence leaves open

The README closes its contents with a section titled what this release is not, which is a useful thing for a project like this to publish and a useful thing for an adopter to read before the first download. That section is worth reading in full before you commit to the 118 GiB download.

The gating constraint is the hardware. The README states the engine runs on exactly one piece of silicon. If you do not have gfx1151, there is no degraded mode to fall back to, and the ROCm device flags in the quickstart (--device /dev/kfd and /dev/dri) are not decoration. The second constraint is model scope: this is Qwen3.8-Flash-Next, or a llama.cpp GGUF of it. A general engine such as llama.cpp, which the README references as the source format for the bring-your-own-GGUF path, is the right tool when you need to load many architectures on many devices; it is simply not competing on this measurement.

The licence is the open question. Repository metadata reports NOASSERTION, and LICENSE.md is the file to read. There is also a THIRD-PARTY-NOTICES.md at the top level, which matters for a project that ships model weights, a tokenizer and a quality sidecar alongside its own code. NOASSERTION means a tool could not classify the licence, not that no licence exists. Read LICENSE.md and THIRD-PARTY-NOTICES.md and decide for your own use case; nothing here is legal advice.

Maintenance posture

The repository is not archived, and the last push was on 2026-09-15. The release referenced throughout the quickstart is 0.11.1, with behaviour changes documented at 0.6.0 and 0.7.0. The CHANGELOG.md at the top level is where the version-to-version detail lives; the README's own notes about the 0.6.0 sidecar change and the 0.7.0 GGUF path show that the changelog is doing real work rather than recording cosmetic bumps.

The upgrade cost is concentrated in two places. The image tag is load-bearing: the compose file's warning about api and engine drift means a partial upgrade is the failure mode to avoid, and the fix is to move both services to the same tag. The sidecar re-fetch is the second: a writable models volume plus HALOGEN_DOWNLOAD set will pull the 2.4 GiB file again when it predates the image, while the 115 GiB checkpoint stays put. Neither is expensive in the ordinary case, and both are cheap to avoid once you know they exist.

Editorial conclusion

Adopt it if you have a Strix Halo machine and want this specific model served over an OpenAI-compatible endpoint with speculative decode that the README says is byte-identical to serial greedy output at temperature 0. Do not adopt it if your GPU is anything other than gfx1151, or if you need a general runtime that loads arbitrary architectures: the README states there is no portability layer and no fallback path. Before committing, verify three things in the repository itself: the license terms in LICENSE.md, which the metadata reports as NOASSERTION, the context and memory guidance before you size a KV pool, and the rule that both containers run the same image tag, since a mismatched api and engine pair was the cause of public issue #26.

Frequently asked questions

Official sources

1. Issues
2. peonist-ai/halogen-flash-server on GitHub
3. Project website
4. README