[libratechw]

Title

Issue: Refs #1258

Summary

With STRATA_PF_FUSED=1 the routed experts of a prompt run on gfx11's int8 matrix cores, each weight sub-block decoded to int8
with a per-block epilogue. STRATA_PF_F16=1 runs the covered formats on the F16 matrix cores instead: the codes stay packed in
LDS, each wave turns its own rows into F16, and the F16 WMMA accumulates all of K in FP32. The K-quant kernels adapt gufo's
RoutedF16GEMMKernel (97b8645f, MIT; attribution and licence text at the top of both new files). The output differs from the
int8 kernels', so it is off by default; gfx12 keeps the int8 kernels.

What changed

• src/prefill/moe_f16_common.cuh (new): the switch, the work item, the activations staged as F16, the WMMA loop, the two
  epilogues.
• src/prefill/moe_f16_kq.cu/.hpp (new): Q4_K / Q5_K gate/up and Q5_1 down (with STRATA_PF_FUSED_KQ=1). They read the
  weights 16 / 8 bytes at a time, so each of gate/up and down takes them only when the batch's blobs, offsets and row
  strides are aligned to that (the native blobs are promised 2 bytes); otherwise that half stays on its int8 kernel. gfx906,
  whose build leaves this file out, covers nothing here.
• src/prefill/moe_fused_iq.cu: native_f16_kernel for IQ2_XXS / IQ2_XS / IQ2_S / IQ3_XXS / IQ3_S / IQ4_XS gate/up and Q2_0 /
  IQ4_NL / Q8_0 down (through the existing 16-bit loads), and the dispatch.
• tests/cuda/prefill_fused_iq_test.cpp, CMakeLists.txt: with STRATA_PF_F16=1 on a gfx11 device the test allows 5e-3
  against the int8 model and MMQ (measured 2.7e-3) and places the blobs 256-byte aligned, so the F16 kernels are what it
  checks; elsewhere the int8 limits stay. ctest hip_prefill_fused_iq_f16 runs the default pairs with the switch,
  hip_prefill_fused_iq_f16_kq the four K-quant pairs (it sets STRATA_PF_FUSED_KQ=1 too).
• docs/STRIX_HALO.md: what it covers, the layer and prompt times, first-token KL.

Ryzen AI Max+ 395 (gfx1151, ROCm 7.14.1, Linux), on 296b59d. One layer, hip_prefill_fused_iq, 16,384 tokens, top 10 over
512 experts, median of 3:

gate/up / down	int8 kernels, ms	F16, ms
Q4_K / Q5_1 (UD-Q4_K_XL)	86.4	53.7 (-38%)
Q4_K / Q8_0	85.9	53.5 (-38%)
IQ3_S / IQ4_NL (UD-IQ4_XS)	79.8	58.2 (-27%)

Prompt tok/s, the branch's binary without and with STRATA_PF_F16=1, --serve --prefill 16384 --spec 4, the section-5
switches with STRATA_PF_SWITCH_MIN_T=2048 (UD-Q4_K_XL also STRATA_PF_FUSED_KQ=1), a chat request of 4K / 32K tokens,
fresh servers alternating, 5 pairs after dropping those with page migration or reclaim; median tok/s and the median of the
pair ratios, every pair faster:

	4K	32K
UD-Q4_K_XL	1,129 -> 1,294 (+14.4%)	1,249 -> 1,422 (+13.9%)
UD-IQ4_XS	1,114 -> 1,269 (+13.5%)	1,266 -> 1,389 (+9.9%)

The UD-Q4_K_XL 32K cell is of this code; the layer times and the other three cells are of an earlier head with the same
kernels (before the alignment check and the test changes). Without the switch, the layer's output hashes (six pairs, 2,048
and 16,384 tokens) are 296b59d's.

First-token KL against a run with no switches, 12 natural-text chat requests of 1.5K-24K tokens, mean (median); the fast arms
with STRATA_PF_SWITCH_MIN_T=0; the top token is the same in 12 of 12 in every arm:

	section-5 switches + int8 fused experts	+ STRATA_PF_F16=1
UD-Q4_K_XL	0.0065 (0.0017)	0.0114 (0.0048)
UD-IQ4_XS	0.0076 (0.0035)	0.0104 (0.0057)

F16 moves further from the default than the int8 path, so this stays opt-in. The records and scripts are in
libratechw/Strata@e5d05554, 2020/.

Builds and tests:

• HIP gfx1151, this head: ctest, 114 tests: 106 pass (the three hip_prefill_fused_iq* among them), 5 skip, and the same
  three fail as on 296b59d (ple_parity, expert_parity, pool_test).
• HIP: moe_fused_iq.cu and moe_f16_kq.cu compile for gfx1012, gfx1030, gfx1100, gfx1103, gfx1150, gfx1151, gfx1200 and
  gfx1201 (77506be1; the head changes neither file). gfx906: moe_fused_iq.cu no longer leaves the f16kq functions
  undefined (it did at 77506be1); the whole gfx906 build stops earlier, at mmq_nvfp4_w4a4.cu (ggml's vendors/cuda.h),
  which this branch does not touch.
• CUDA 13.0, RTX 3060 (Windows), 77506be1 (the head's changes since are in a header CUDA does not include, a HIP ctest, a
  test comment and the doc): builds; ctest, 117 tests: 114 pass, 2 skip, and ple_parity fails as on 296b59d.
• SYCL: not affected (the SYCL experts are their own files).

Not run: other gfx11 cards (gfx1100-1103, gfx1150), and quality beyond the first-token KL.

🤖 Generated with Claude Code

[Niko1221]

Ported unchanged (all seven commits, your authorship), as an opt-in: STRATA_PF_F16=1 (and STRATA_PF_F16_KQ=1 with STRATA_PF_FUSED_KQ=1 for K-quants). We have no gfx11 matrix-core machine, so we only checked what we can: it builds for gfx1100/1103/1150/1151/1201, gfx12 keeps the int8 kernels (hip_prefill_fused_iq in both modes passes on an R9700) and the identity gate there is unchanged. The speed and KL numbers are yours; a larger prompt set and a task check from a gfx1151 would let us think about a default.

[libratechw]

Thanks for porting this. You asked for a larger prompt set and a task check on gfx1151. I ran both on the merged build (477ac13). B is F16 off, F is F16 on (STRATA_PF_F16=1 with STRATA_PF_FUSED_KQ=1; I could not find STRATA_PF_F16_KQ in the current source). R is the reference.

KL vs R, 25 prompts, first-token logits

model	arm	KL mean	KL max	top-1
UD-Q4_K_XL	B	0.00788	0.0472	24/25
UD-Q4_K_XL	F	0.00834	0.0489	24/25
UD-IQ4_XS	B	0.00681	0.0454	25/25
UD-IQ4_XS	F	0.00394	0.0393	25/25

Task check (greedy, reasoning off, one run per arm): HumanEval every second problem (82, EvalPlus 0.3.1) and 30 generated reasoning tasks.

model	arm	HumanEval base	HumanEval plus	reasoning
UD-Q4_K_XL	B	80/82	78/82	30/30
UD-Q4_K_XL	F	81/82	79/82	30/30
UD-IQ4_XS	B	77/82	75/82	30/30
UD-IQ4_XS	F	77/82	74/82	30/30

F16 was engaged: 16–17 of the 82 HumanEval responses differ between B and F. The only problem whose result flips is HumanEval/76 (F passes on Q4_K_XL, fails on IQ4_XS), so I see no one-directional degradation. The reasoning outputs are identical between B and F (0 of 30 differ), probably because those prompts are short and do not reach the F16 path, so they say nothing about F16. Each arm was run once, so I have no noise floor for single-problem differences.

Per-prompt KL, per-problem grading, all responses, settings and scripts (klrun.py, taskcheck.py): https://github.com/libratechw/Strata/tree/09df26f072914f96cfa4fcae3e835e462358808e/main477/2020

[Niko1221]

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.