Skip to content

tests: retire the #450 known-fail guard, the symmetric-i4 route is not a regression (engine #450) - #109

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/hrx-test-450-retire
Oct 18, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/hrx-test-450-retire

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Retires the known_fail("#450") guard: engine ggml-org#450's accepted regression is not a regression on AMD's current kernels, so the case states what ships instead of asserting the route is never selected.

Measurement (strixhalo, gfx1151)

Qwen3-4B-Q5_K_M, route on vs off (GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32), everything else identical, -dev HRX0 -ngl 99 -t 4 under ~/lb/thermal-run.sh:

route on route off
pp512 3033.38 ± 23.60 2991.08 ± 39.41
pp2048 2647.52 ± 95.31 2613.31 ± 117.61
tg64 65.65 ± 0.36 63.61 ± 0.11

Qwen3-4B-IQ4_XS, the second qualifying quant: pp512 2933 vs 2971, pp2048 2617 vs 2573, tg64 73.70 ± 0.15 vs 73.00 ± 0.46 — mixed, inside the noise.

Numerics on held-out natural text (Gutenberg Pride and Prejudice + Moby Dick, 1.97 M chars — not the repetitive text that made the first run meaningless):

  • PPL = 16.3451 ± 0.74733 over 20 chunks at -c 512 (arm A, logits saved);
  • cross-arm llama-perplexity KLD over 5 chunks: Mean 0.000000 ± 0.000000, maximum 0.000056, max Δp 0.005 %;
  • three 64-token greedy completions at temperature 0: byte-identical between the arms (931 bytes each).

Device use was confirmed, not assumed: the runs show dev HRX0 / ngl 99 and reach pp512 ≈ 3000 tok/s on a 4B against 89.76 tok/s for the same model at -ngl 0.

What the case asserts now

symmetric_i4_consumer_qualification (re-added by #105 with its pre-#104 expectations) asserted the symmetric-i4 producer is never selected. It now asserts the truth: a qualifying graph — a Q5_K/IQ4_XS weight consuming the normalised value, directly or through a reshape — selects loom_libs:ggml_rmsnorm_binary_symmetric_i4_k32, and a graph with no such consumer does not.

Its q8-publication assertions are removed from this case on purpose: ggml_rmsnorm_binary_q8_1_x4_publish went with f2099e9b, so that is engine ggml-org#449's territory and stays with the guards there rather than failing a route case for a publication reason.

Gate (the fork has no CI)

Local host suite on this base (29dd8820, CPU only, no HRX device):

host: 66 case(s)
host: 47 passed, 13 failed, 6 known-fail, 0 unexpected-pass, 66 total

symmetric_i4_consumer_qualification is now [PASS]; the 6 remaining known-fail rows are ggml-org#449's; the 13 [FAIL] rows are the class A plan-shape, class B kernel-name and the qwen_router_top8_dispatch NEEDS-DEVICE rows, unchanged.

Related: engine ggml-org#450 (this ruling, with the numbers), ggml-org#445 (the classification), ggml-org#449 (publication).

🤖 Generated with Claude Code

…te is not a regression (engine ggml-org#450)

Measured on strixhalo (gfx1151) with Qwen3-4B-Q5_K_M, route on vs off
(GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32), everything else identical,
-dev HRX0 -ngl 99 -t 4:

  pp512   3033.38 +- 23.60   vs  2991.08 +- 39.41   (route on is faster)
  pp2048  2647.52 +- 95.31   vs  2613.31 +- 117.61
  tg64      65.65 +-  0.36   vs    63.61 +-  0.11   (+3.2 %, far outside the error bars)

Numerics, held-out Gutenberg text (Pride and Prejudice + Moby Dick, replacing the repetitive text
that made the first run meaningless): PPL = 16.3451 +- 0.74733 over 20 chunks at -c 512, cross-arm
llama-perplexity KLD over 5 chunks 0.000000 +- 0.000000 (maximum 0.000056, max delta-p 0.005 %), and
three 64-token greedy completions at temperature 0 byte-identical between the arms (931 bytes each).
f2099e9's "substantial numerical performance regression" is not observable on AMD's current kernels.

So symmetric_i4_consumer_qualification states what ships rather than what was hoped: a qualifying
graph (a Q5_K/IQ4_XS weight consuming the normalised value, directly or through a reshape) selects
loom_libs:ggml_rmsnorm_binary_symmetric_i4_k32, and a graph with no such consumer does not. Its old
q8-publication assertions are engine ggml-org#449's territory (ggml_rmsnorm_binary_q8_1_x4_publish went with
f2099e9) and stay with the guards there instead of failing this case for a publication reason.

Host suite on this base: 66 cases - 47 passed, 13 failed, 6 known-fail, 0 unexpected-pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant