[dzannotti]

Ports the ROCmFPX tensor formats from ciru-ai/ROCmFPX (MIT) onto current master additively — nothing upstream is removed.

ggml-cuda: 0 files modified. The HIP backend still builds unchanged, which is what makes this safe to carry and to sync.

57 files, +9,909/−762.

Types

On ciru's numbering, so their published GGUFs load correctly:

id	type	bpw
100	Q4_0_ROCMFP4	4.50 — UE4M3 scales + packed AMD FP4 blocks
101	Q4_0_ROCMFP4_FAST	4.25 — single-scale speed layout
102	Q6_0_ROCMFPX	6.50 — signed-magnitude integer levels
103	Q8_0_ROCMFPX	8.25 — signed int8, clamped
104	Q3_0_ROCMFPX	3.50
105	TURBO3_0	3.50 — KV cache
106	TURBO4_0	4.50 — KV cache
107	Q7_0_ROCMFPX	7.50 — DualView, packed signed-Q7 + FP16 group scales
108	Q2_0_ROCMFPX	2.50 — S40 codebook + dual UE4M3 scales

Every type ciru adds over upstream, verified by set comparison. GGML_TYPE_COUNT = 109 matches ciru exactly.

Numbering: these are on-disk GGUF ids. Laurent's fork independently assigned 107 = Q2_0_ROCMFPX, colliding with ciru's 107 = Q7_0_ROCMFPX. This follows ciru. Ids 100–104 agree under both, so all FP4/FP6/FP8 models are unaffected either way.

What is here

Format definitions and reference math (ggml/rocmfpx, ggml/rocmfp4), CPU paths, Vulkan dequant shaders and integer-dot (MMQ/MMVQ) paths, gguf-py support, llama-quantize recipes.

What is not here yet — and why it is a phase, not an omission

The ROCm/HIP kernels.

ciru's ggml-cuda is not additive on top of upstream — it replaces upstream machinery. In mmq.cuh, ciru's own base carried 8 mmq-config-*.cuh includes and 42 ggml_cuda_mmq_get_config references; ciru's version has neither, substituting the older tile_x_sizes / MMQ_DP4A_TXS_* scheme. Merging it reverts upstream's per-arch MMQ configuration for every quant type and every GPU, and leaves 16k lines of divergence in a directory upstream touches ~81 times per 90 days.

Upstream has since restructured MMQ into an extension-friendly shape:

mmq-load-tiles.cuh                        load_tiles_<type>
mmq-vec-dot.cuh                           vec_dot_<type>
template-instances/mmq-instance-<type>.cu DECL_MMQ_CASE(GGML_TYPE_...)
mmq-config-rdna3-5.cuh                    CASE(...) entries — RDNA3.5 is gfx1151

with NVFP4 and MXFP4 already shipping through it. Phase 2 re-derives ciru's kernels onto that path, per type, so they add rather than replace — keeping NVIDIA paths intact (not because they are used here, but because deleting them is what makes a fork unmergeable) and keeping future syncs clean.

The wholesale port is preserved on rocmfpx/wholesale-reference — it builds on both backends (Vulkan 663 targets, HIP ggml-hip + llama-quantize, 0 errors) and is the reference for the re-derivation. It is not proposed for merge.

A behavioural change to be aware of

The FP6 codebook's first negative entry is corrected to -32 from -0. The fork's own port notes record the old value disagreeing with rocmfpx_decode_fp6_code and costing ~2% error. This changes FP6 decode output.

Verification

Vulkan (-DGGML_VULKAN=ON, fedora 44)	663 targets, llama-cli + llama-quantize linked
HIP (-DGGML_HIP=ON -DGPU_TARGETS=gfx1151)	builds green with ggml-cuda untouched

Compile and link only. Nothing ran on hardware, no model was quantized or evaluated. Before this drives anything it wants test-backend-ops on the GPU and a perplexity run against a known-good quant — the FP6 change especially.

For context on whether this is worth landing: the fork's measurements put Q4_0_ROCMFP4 at 14.63 GiB / 12.56 tg128 / 5.9842 PPL against UD-Q4_K_XL at 16.69 GiB / 11.71 / 5.8965 — smaller and faster, marginally worse quality, error bars overlapping. No comparison against IQ4_XS exists in either project, which is arguably the incumbent it has to beat.

[dzannotti]

@LaurentZuijdwijk i think we should close this in favor of the work you was already doing