ROCmFPX: AMD-first GGUF weight formats for llama.cpp

What ROCmFPX adds to llama.cpp and who it is for

ROCmFPX is not a standalone inference engine. It is a fork of llama.cpp that adds experimental AMD-focused GGUF model-weight formats named ROCmFP2, ROCmFP3, ROCmFP4, ROCmFP6 and ROCmFP8, with CPU reference implementations backing accelerated HIP/ROCm and Vulkan kernels. The README is explicit that these are real weight quants rather than K/V-cache compression, which matters because a lot of AMD-oriented work in this space only touches the cache.

The audience is narrow and technical: people who already build llama.cpp from source, run AMD GPUs, and are willing to quantize models themselves. The README's own status line calls ROCmFPX an experimental feature family on the canonical main branch, and warns that APIs, tuning choices and performance can change. There are no retrieved releases, so there is no versioned artifact to pin against. If you need a supported, tagged build, this repository is not that.

The repository is a full llama.cpp tree, not a patch set. Top-level entries include ggml/, src/, common/, tools/, examples/, conversion/, gguf-py/, CMakeLists.txt and CMakePresets.json. That layout means adopting ROCmFPX means adopting a fork of the whole build, with the maintenance cost that implies.

The ROCmFP2 block layout and the S40 codebook

The mechanism that separates ROCmFP2 from generic 2-bit schemes is the block layout. According to the developer update, ROCmFP2 ships as Q2_0_ROCMFPX and uses a 2.50-bpw block layout with an S40 {-4, -1, +1, +4} codebook and dual UE4M3 scales. In plain terms: each block stores a small set of signed levels, and two scales in UE4M3 format accompany the block so that different parts of the tensor can be rescaled independently. The README states that the smaller blocks reduce model storage and memory traffic compared with ROCmFP4.

The 2.50-bpw number needs care. The README notes that it describes native ROCmFP2 weight blocks, and that complete GGUF BPW can be higher because files also contain metadata and tensors stored in other types. So the figure is a property of the quantized tensors, not of the file you download.

ROCmFP4 is the speed-first 4-bit family, and the README states that existing Qwen comparisons put ROCmFP4 files about 12% below the matched Q4_K_M size. ROCmFP3, ROCmFP6 and ROCmFP8 are listed as development preview: they are in the tree, but kernel optimization, routing and model coverage are still being improved. Treating all five formats as equally usable would misread the project's own table.

Which format to pick, and what the published throughput shows

The README's status table splits the family in two. ROCmFP2 and ROCmFP4 are marked optimized and validated, performance-tuned and benchmarked on the tested Strix Halo Vulkan and HIP/ROCm paths. ROCmFP3, ROCmFP6 and ROCmFP8 are development preview. That split is the single most useful fact for a decision, because it tells you where the tuning effort has gone.

For matched Qwen3.6-35B-A3B tests, the README reports a 256-token, non-speculative decode workload from an internal NVMe. On Vulkan0, ROCmFP2 reached 90.30 tok/s against 76.20 tok/s for ROCmFP4 STRIX_LEAN, an 18.50% advantage. On ROCm0 with ROCm 7.14, the same comparison was 75.90 tok/s against 67.50 tok/s, or 12.44%. The README frames these as demonstrating the benefit on the tested system, not a universal speed or quality guarantee.

The quality caveat is stated directly: ROCmFP2 is more lossy than ROCmFP4, so important workloads should be compared against the BF16/F16 source. That is the trade you are making. ROCmFP2 buys size and decode speed; ROCmFP4 buys fidelity. The README does not publish a perplexity or task-accuracy table for either, so the lossiness claim has no numeric counterpart in the published tables.

Building ROCmFPX and running a first model

There is no release artifact. The README points to the current main source ZIP and to a clone command in its Quick Start section, which is titled Quick Start (Strix Halo / gfx1151). The README gives the clone step directly:

git clone https://github.com/charlie12345/ROCmFPX

Because the tree contains CMakeLists.txt and CMakePresets.json, configuration goes through CMake. The README does not print a full build command, and the repository's Makefile has been replaced: it now aborts with a message stating that the Makefile build has been replaced by CMake and pointing to https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md. So the build instructions live in that upstream document plus the presets in this tree, and you should read both rather than guess at flags.

Once a binary exists, the README's MTP results describe the invocation shape used for its own measurements: llama-cli with full GPU offload, FlashAttention, -c 4096, greedy sampling (--temp 0), and -b 512 -ub 512. The README reports that throughput is the final Generation: rate from llama-cli. If you are comparing formats, keep the prompt and token count fixed, because the published MTP numbers are matched pairs and the README notes that gains are content-dependent.

MTP speculative decoding without a separate draft model

One of the more distinctive features is built-in MTP acceleration. Models with an MTP/NextN head, including M-RoPE Qwen models, can use target-verified self-speculative decoding without loading a separate draft model. That removes the memory cost and the vocabulary-matching problem that come with a conventional draft model setup.

The README publishes local llama-cli decode results from the promoted main source on Strix Halo (gfx1151). Qwable-5-27B-Coder ROCmFP4 COHERENT_AGENT on Vulkan0 went from 14.0 t/s to 33.2 t/s with an n6 / p0.60 profile, a 2.37x speedup on a matched 64-token pair. Qwen3.6-35B-A3B ROCmFP4 STRIX_LEAN-FRESH on Vulkan0 went from 76.5 t/s to a 116.1 t/s median and 118.3 t/s peak with n4 / p0.55, a 1.52x median speedup over three matched 256-token runs. The same model on ROCm0 reached a 106.2 t/s median with MTP; the README shows no no-MTP baseline for that row.

The honest reading is in the README itself: MTP gains are content-dependent, and predictable code, JSON and lists usually accept more draft tokens than creative prose. The n4 / p0.55 and n6 / p0.60 profiles are tested starting points, not defaults. There is also a promotion comparison: the promoted source and the pre-promotion experimental build were effectively tied on Qwen3.6, with median differences of -0.7% on Vulkan and -0.3% on ROCm, and a longer 512-token Vulkan run reached 110.7 t/s against 107.2 t/s.

The gfx1151-only ROCmI4 W4A4 path and its limits

ROCmFPX includes an opt-in HIP/ROCm path for Q4_0_ROCMI4 models that keeps ROCmI4 weights packed as signed four-bit values and uses the native v_wmma_i32_16x16x16_iu4 instruction for batched matrix multiplication. The README describes the intended scope as prompt processing and MTP target verification; ordinary one-token-at-a-time generation continues to use MMVQ. So this is not a general decode accelerator.

The gating is deliberately tight, and the README lists the conditions. GGML_HIP_ROCMI4_W4A4 defaults to OFF. Device code is compiled only under the exact __gfx1151__ target. Runtime dispatch also requires the exact gfx1151 device identifier. Other GPUs and default builds fall back to the existing exact int8 MMQ path, and no backend-test tolerance is weakened when W4A4 is enabled.

That last point is worth pausing on. A four-bit weight, four-bit activation path is exactly the kind of change where it is tempting to loosen a test tolerance to make it pass. The README states that did not happen, which is a stronger correctness signal than a throughput number. The cost is portability: if your GPU is not gfx1151, this feature is simply unavailable to you, and the README does not claim a roadmap for other targets. The ROCmI4 terminology table also separates ROCmI4 (a GGUF model-weight format) from INT4 (a generic integer width), which is worth keeping straight when reading the code.

Alternatives, licensing and what to check before adopting

The obvious alternative is upstream llama.cpp with its own quantization types. The difference in approach is not just the format list. Upstream ships tagged releases and a documented build path; ROCmFPX is a fork on main with no retrieved releases, and the README states that older tags and previously built binaries do not receive these updates automatically. Upstream's K-quants such as Q4_K_M are widely converted and widely tested; ROCmFPX's formats require this fork to load at all. If you want portability across backends and tooling, upstream wins on that axis. If you specifically want smaller AMD-targeted weight formats plus the MTP and ROCmI4 work, that is what the fork exists for.

On licensing, the repository is MIT and includes LICENSE, THIRD_PARTY_NOTICES.md and a licenses/ directory. Since ROCmFPX is a llama.cpp fork, the third-party notices file is the place to look for inherited components. That is a pointer, not legal advice; if you redistribute binaries, read those files yourself.

Upgrade cost is the real ongoing expense. Because there are no releases to pin, updating means pulling main and rebuilding, and the README warns that APIs, tuning choices and performance can change. The README also carries a disclosure that AMD provided the development hardware under a creator partnership, and states that all development, benchmarks, code, testing, documentation and opinions in the repository are the author's own. That is a transparency statement, not a defect, but it is context for reading the benchmark tables.

Editorial conclusion

Adopt ROCmFPX if you already build llama.cpp from source and run AMD hardware, start with ROCmFP4 on a BF16/F16 source, and verify quality on your own task before replacing a Q4_K_M or Q8_0 setup. Avoid it if you need packaged binaries, non-AMD GPUs, or stable APIs, because the README calls the whole family experimental and states that older tags and previously built binaries do not receive the updates automatically. Before committing, check that your device identifier is gfx1151 if you want the ROCmI4 W4A4 path, since that device code is compiled only under the exact __gfx1151__ target.

Frequently asked questions

Official sources

1. charlie12345/ROCmFPX on GitHub
2. Issues
3. License: MIT
4. README