Repository navigation
tests: retire the #450 known-fail guard, the symmetric-i4 route is not a regression (engine #450) - #109
Merged
bong-water-water-bong merged 1 commit intoOct 18, 2026
Conversation
…te is not a regression (engine ggml-org#450) Measured on strixhalo (gfx1151) with Qwen3-4B-Q5_K_M, route on vs off (GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32), everything else identical, -dev HRX0 -ngl 99 -t 4: pp512 3033.38 +- 23.60 vs 2991.08 +- 39.41 (route on is faster) pp2048 2647.52 +- 95.31 vs 2613.31 +- 117.61 tg64 65.65 +- 0.36 vs 63.61 +- 0.11 (+3.2 %, far outside the error bars) Numerics, held-out Gutenberg text (Pride and Prejudice + Moby Dick, replacing the repetitive text that made the first run meaningless): PPL = 16.3451 +- 0.74733 over 20 chunks at -c 512, cross-arm llama-perplexity KLD over 5 chunks 0.000000 +- 0.000000 (maximum 0.000056, max delta-p 0.005 %), and three 64-token greedy completions at temperature 0 byte-identical between the arms (931 bytes each). f2099e9's "substantial numerical performance regression" is not observable on AMD's current kernels. So symmetric_i4_consumer_qualification states what ships rather than what was hoped: a qualifying graph (a Q5_K/IQ4_XS weight consuming the normalised value, directly or through a reshape) selects loom_libs:ggml_rmsnorm_binary_symmetric_i4_k32, and a graph with no such consumer does not. Its old q8-publication assertions are engine ggml-org#449's territory (ggml_rmsnorm_binary_q8_1_x4_publish went with f2099e9) and stay with the guards there instead of failing this case for a publication reason. Host suite on this base: 66 cases - 47 passed, 13 failed, 6 known-fail, 0 unexpected-pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retires the
known_fail("#450")guard: engine ggml-org#450's accepted regression is not a regression on AMD's current kernels, so the case states what ships instead of asserting the route is never selected.Measurement (strixhalo, gfx1151)
Qwen3-4B-Q5_K_M, route on vs off (
GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32), everything else identical,-dev HRX0 -ngl 99 -t 4under~/lb/thermal-run.sh:Qwen3-4B-IQ4_XS, the second qualifying quant: pp512 2933 vs 2971, pp2048 2617 vs 2573, tg64 73.70 ± 0.15 vs 73.00 ± 0.46 — mixed, inside the noise.
Numerics on held-out natural text (Gutenberg Pride and Prejudice + Moby Dick, 1.97 M chars — not the repetitive text that made the first run meaningless):
-c 512(arm A, logits saved);llama-perplexityKLD over 5 chunks: Mean 0.000000 ± 0.000000, maximum 0.000056, max Δp 0.005 %;Device use was confirmed, not assumed: the runs show
dev HRX0 / ngl 99and reach pp512 ≈ 3000 tok/s on a 4B against 89.76 tok/s for the same model at-ngl 0.What the case asserts now
symmetric_i4_consumer_qualification(re-added by #105 with its pre-#104 expectations) asserted the symmetric-i4 producer is never selected. It now asserts the truth: a qualifying graph — a Q5_K/IQ4_XS weight consuming the normalised value, directly or through a reshape — selectsloom_libs:ggml_rmsnorm_binary_symmetric_i4_k32, and a graph with no such consumer does not.Its q8-publication assertions are removed from this case on purpose:
ggml_rmsnorm_binary_q8_1_x4_publishwent withf2099e9b, so that is engine ggml-org#449's territory and stays with the guards there rather than failing a route case for a publication reason.Gate (the fork has no CI)
Local host suite on this base (
29dd8820, CPU only, no HRX device):symmetric_i4_consumer_qualificationis now[PASS]; the 6 remainingknown-failrows are ggml-org#449's; the 13[FAIL]rows are the class A plan-shape, class B kernel-name and theqwen_router_top8_dispatchNEEDS-DEVICE rows, unchanged.Related: engine ggml-org#450 (this ruling, with the numbers), ggml-org#445 (the classification), ggml-org#449 (publication).
🤖 Generated with Claude Code