Repository navigation
hrx: the fused rmsnorm_gate -> packed-quantised matmul pair returns garbage for every output row (Q4_K and Q6_K, deterministic, model-free reproducer) #969
Description
hrx: the fused rmsnorm_gate -> packed-quantised matmul pair returns garbage for every output row (Q4_K and Q6_K, deterministic, model-free reproducer)
Summary
A five-node graph (rms_norm -> mul(gamma) -> sigmoid -> mul -> mul_mat with a quantised weight) is dispatched as the
fused pair
loom_libs:ggml_rmsnorm_gate_f32_q8_1_x4 writes a Q8_1 activation into a transient
loom_libs:ggml_mul_mat_f32_f32_wmma reads it (activation_format=Q8_1) with weight_format=46 (Q6KRow64)
and the weight bound in the packed q6k-packed-k256-row64-scalerow layout
and the matmul output is wrong for every one of the 64 weight rows: 35 rows NaN, 29 rows finite garbage ~1e8 x the
reference (max|err|/max|ref| = 5.2e8). CPU reference of the same graph is correct.
Same graph, same weights, same leaves, one difference — the producer's output has one extra consumer — and the plan
becomes ggml_rmsnorm_gate_f32_f16 -> ggml_kquant_mul_mat_decode_f32 (native weight) and the result is exact
(max|err|/max|ref| = 7.2e-3, the f16 level). So the defect is in the q8_1-activation + packed-weight pair, not in the
matmul arithmetic and not in the graph.
It is deterministic (two processes, byte-identical output), it is not weight-type specific (Q4_K fails the same
way with weight_format=44 Q4KRow64), and it is not stale/aliased weight memory: with every weight byte set to zero
through the backend, the output is exactly 0 for all 2560 values on all three arms.
Reproducer (no model, ~1 s of GPU, licence headers in both files)
# binary: llama.cpp fork, base d8bd0ff60 + the top_k branch only (my branch touches hip_top_k_f32.hip and nothing else)
# ~/wt/fn-topk/build/bin , libggml-hrx.so.0.18.0
~/wt/hrx-probe/kquant_probe ngated # prints arm=... MATCH|DIVERGES + a per-residue row map
~/lb/tattn-kquant-split.sh # Q6_K/Q4_K x tiled/distinct rows, repeat arm for determinism
~/lb/tattn-kquant-zero.sh # zero-weight control
~/lb/tattn-kquant-dump.sh # GGML_HRX_DUMP_COMMAND_PROGRAM_DIR for the diverging + matching plans
kquant_probe builds exactly the GDN layer's closing projection at the model's real shapes (head_dim 128 x 48 v-heads =
6144 in, 2560 out, 1 token) and walks the producer chain back to a leaf: leaf, reshape, sig, rms, rms_nv,
ngated, ngated_nv. CPU is the reference, same binary, same leaves.
Measured (~/lb/aval/attn-kquant-split/20261104T060418Z, ~/lb/aval/attn-kquant-zero/20261104T062132Z)
| arm | producer -> consumer dispatch | weight | nonfinite | max|err|/max|ref| | verdict |
|---|---|---|---|---|---|
| leaf | kquant.mul_mat.decode_f32 |
Q6_K | 0 | 3.0e-3 | MATCH |
| reshape | kquant.mul_mat.decode_f32 |
Q6_K | 0 | 3.1e-3 | MATCH |
| sig | kquant.mul_mat.decode_f32 |
Q6_K | 0 | 2.2e-3 | MATCH |
| rms / rms_nv | common.rmsnorm_binary_f32 -> kquant.mul_mat.decode_f32 |
Q6_K | 0 | 6.1e-3 | MATCH |
| ngated | common.rmsnorm_gate -> common.mul_mat.f32_f32_decode |
Q6_K | 1400 / 2560 | 5.2e+8 | DIVERGES |
| ngated_nv | same | Q6_K | 1400 / 2560 | 5.9e+8 | DIVERGES |
| ngated | same | Q4_K | 1920 / 2560 | 1.2e+8 | DIVERGES |
| ngated, distinct rows | same | Q6_K | 1300 / 2560 | 6.8e+8 | DIVERGES |
| ngated, distinct rows | same | Q4_K | 1708 / 2560 | 1.4e+8 | DIVERGES |
| ngated (f32 weight) | common.mul_mat.f32_f32_decode |
f32 | 0 | 1.7e-4 | MATCH |
| ngated + one extra consumer of the producer output | common.rmsnorm_gate -> kquant.mul_mat.decode_f32 |
Q6_K | 0 | 7.2e-3 | MATCH |
| ngated, every weight byte 0 | common.rmsnorm_gate -> common.mul_mat.f32_f32_decode |
Q6_K / Q4_K | 0 | out == 0 exactly | MATCH |
Not a subset defect: the 29 "finite" rows are not correct either - their values are -5.0e7, -3.8e8, +1.0e8, i.e.
~1e8 x the reference. Per residue class (out[j] depends on w[j % 64] because the probe tiles 64 distinct rows), 35 of
the 64 classes are NaN and the other 29 are huge garbage: all 64 output rows are wrong.
The command programs (dumped, diffable)
~/lb/aval/attn-kquant-dump/20261104T060144Z/
diverging (stage0-ngated/dump/program-0-shape-04bf226ae8557ff2/program.json), commands=3:
0 ggml_rmsnorm_gate_f32_q8_1_x4 ip:token_count=48 hidden_size=128
b: output:6 (Transient, 24576) b: q8_output:12 (Transient, 6912 = 192 blocks x 36 B)
1 ggml_mul_mat_f32_f32_wmma cp: activation_format=9 (Q8_1) cp: weight_format=46 (Q6KRow64)
cp: input_size=6144 output_size=2560 ip: token_count=1
b: input:12 (Transient, 6912) b: weight:8 (q6k-packed-k256-row64-scalerow [14,6144,2560])
arena=31488 transients t:6@0:24576 t:12@24576:6912 t:9@0:10240
matching (stage1-ngated/.../program-0-shape-81430fc9f7fabca3/program.json), commands=4:
0 ggml_rmsnorm_gate_f32_f16 b: output:6 (24576) b: f16_output:14 (12288)
1 ggml_kquant_mul_mat_decode_f32 cp: weight_format=6 (Q6K) b: weight:8 (ggml-native)
Two things worth a look first, both in the diverging program: the producer is specialised with
token_count=48, hidden_size=128 (48 heads x 128) while the consumer sees a 1-token, 6144-wide activation, and
the activation it hands over is the x4 Q8_1 variant. So the q8_1 buffer that the consumer reads is written 48 rows of
4 blocks each and read as 192 consecutive blocks (byte counts agree: 48 x 144 = 192 x 36 = 6912), which is only correct
if _x4 is exactly plain q8_1 row-major. Layout or coverage of that hand-off is the first suspect; the second is the
activation_format=9 read inside ggml_mul_mat_f32_f32_wmma
(kernel-corpus/kernels/loom-libs/motifs/mul_mat_f32_f32_wmma_core.loom, @ggml_q8_lowtoken_geometry).
Why this could not be found by a model run (and how to see it in one)
The pair only forms when a fused gated-norm produces the matmul's activation: in the same probe, the ordinary
rms_norm -> mul(gamma) -> matmul chain takes common.rmsnorm_binary_f32 -> kquant.mul_mat.decode_f32 and is exact,
which is what every non-gated transformer layer does. The gated norm appears in the GDN / linear-attention layers
(src/models/qwen4exp.cpp), i.e. exactly the layers of Qwen3.8-Flash-Next (36 of its 48), whose ssm_out projection is
Q6_K in the shipped UD-IQ3_XXS file. Cheap check on any GDN model: GGML_HRX_LOG_DISPATCH=1 and look for
common.rmsnorm_gate paired with common.mul_mat.f32_f32_decode / ggml_mul_mat_f32_f32_wmma, or dump programs with
GGML_HRX_DUMP_COMMAND_PROGRAM_DIR and grep for activation_format=9 together with weight_format=46|44.
Note also that the gate producer's Q8_1 output has exactly one consumer implementation in the tree: the gate matcher
requires packed_input->ne[1] <= 5 for q8_output, so the prefill branch of is_packed_q8_consumer (Q5_K/IQ4_XS at
256..2048 tokens) can never follow a gate producer - measured: the same graph at 256 tokens quietly takes the f16
producer + hip.dense_prefill.mm and is exact (6.6e-3). So there is no second consumer to cross-check against inside the
engine, which is why this pair can be broken without any other test noticing.
Reachability on a real Flash-Next load is being measured now (short context, GTT-capped); I will post the verdict as a
comment rather than guessing here.
Guards involved, for whoever fixes it
dispatch-rmsnorm.cpp:126is_packed_q8_consumer()— thedecodebranch admitsweight->type == GGML_TYPE_Q6_K
(with a consumer on the matmul output) andQ4_K, which is what lets the gate producer hand a q8_1 activation to a
quantised matmul at all.dispatch-rmsnorm.cpp:799q8_output = ... packed_input->ne[1] <= 5 && is_packed_q8_consumer(...)— picks the
q8_1_x4producer specialization and registers the Q8_1 alternate.dispatch-mul-mat.cpp:1332match_decode_mul_mat_dispatch()(priority 60, the lowest of the mul_mat matchers) —
upgrades tokMulMatF32F32WmmaKerneland callsbuild_mul_mat_dispatch, which finds the existing Q8_1 alternate
(common_prepare_q8_1_x4_input, ExistingAlternateOnly), setspack_q6 = true,activation_format=Q8_1and
weight_format=46. So the lowest-priority matcher is the one that ends up owning a quantified weight.
Not claimed
I have not isolated producer vs consumer to a single kernel yet, and I have not run this against a release binary (the
reproducer ran on ~/wt/fn-topk/build/bin, whose only difference from its base d8bd0ff60 is hip_top_k_f32.hip, so
the rmsnorm/mul-mat path in it is base code). No fix is proposed here.
Activity
bong-water-water-bong commentedon Nov 4, 2026
Correction to my own wording in the zero-weight control above, so nobody builds on an over-claim.
What I measured: with every weight byte set to 0 through the backend (readback fnv identical CPU/HRX, 0 nonzero bytes) the
matmul returns exactly 0 for all 2560 values. What that supports: the plan is not reading uninitialised weight memory,
because the value that comes back is the scaled sum of a zero matrix rather than noise or NaN.
What it does NOT support: that the consumer reads exactly the tensor I wrote. A stale-but-zeroed buffer would give the same
answer, so "the weight binding is the right tensor" is not established by this control. The sentence "So the kernel IS
reading this tensor's bytes" should be read as "the result is consistent with a zeroed weight buffer", no stronger.
Everything else in the issue stands as measured: deterministic byte-identical repeats, both Q4_K and Q6_K, all 64 output
rows wrong (35 NaN, 29 huge garbage ~1e8x the reference), exact match when the same graph routes the matmul to
kquant.mul_mat.decode_f32with the native weight, and the dumped command programs for both plans.bong-water-water-bong commentedon Nov 4, 2026
A second, independent reproducer shows the same defect, and it is not one matcher — it is the whole packed low-token
family (
build_mul_mat_dispatchandbuild_mul_mat_postops_dispatch, both of which setactivation_format=<Q8_1>+weight_format=44/46when they find a pre-quantised activation alternate).The 44-node GDN-layer probe from last night (
~/wt/hrx-probe/gdn_layer.cpp, real Qwen3.8-Flash-Next GDN shapes,head_dim 128 x 48 v-heads, ssm_out Q6_K) has the same signature as the 5-node reproducer in the issue:
~/lb/aval/attn-gdn-layer/20261103T150652Zcommon.mul_mat.f32_f32_decode+common.mul_mat_postops.f32_f32_wmma+common.mul_mat.q6_k_token1_final_projection_q8~/lb/aval/attn-gdn-layer/20261103T160611Zcommon.mul_mat.f32_f32_decode+common.mul_mat_postops.f32_f32_wmma~/lb/aval/attn-gdn-wf32/20261104T002040Z(same graph, f32 weights)common.mul_mat.f32_f32_decodeonly (no packing)So the 1400/2560 count and the +-2e8 magnitude are identical across a 5-node graph and a 44-node GDN graph, and the
element-wise bisect I did last night (
~/lb/aval/attn-gdn-stage/20261104T001032Z, stage dump) shows every upstream stagecorrect: conv_out / v_conv / state_in / normed / ngated / final all match CPU at 3-6e-3 (f16 level, 0 nonfinite), and only
the closing
mul_matis garbage. That is the same story as the issue: the fused gated norm is fine, the quantisedmatmul that consumes its output is not.
Two notes for the fixer:
build_mul_mat_dispatch/build_mul_mat_postops_dispatch/common_prepare_q8_1_x4_input, not in one matcher.same kernel with the native weight binding (an extra consumer on the producer output forces the f16 producer +
kquant.mul_mat.decode_f32) is exact, and an f32 weight is exact, so it is the pre-quantised activation + packedweight combination, not the matvec.
bong-water-water-bong commentedon Nov 4, 2026
Model-level reachability, first answer — and it is the good direction for this issue, with two caveats stated at the end.
I loaded Qwen3.8-Flash-Next UD-IQ3_XXS at
-c 2048withGGML_HRX_DUMP_COMMAND_PROGRAM_DIR(binary = this branch's build,i.e. base
d8bd0ff60+ top_k only) and asked the dumped command program what actually consumes the fused gate producer'sQ8_1 output.
In the real graph the closing GDN projection is fused into the GDN's own HIP kernel, so the broken Loom pair never forms.
loom_libs:ggml_rmsnorm_gate_f32_q8_1_x4appears 36 times (one per GDN layer), andloom_libs:llm_gated_delta_net_f32_wmma_head128_projection_epilogueappears 36 times.hip:k_f69572a806c8with{has_add: 0, input_size: 6144, output_size: 2560, token_count: 2}against the Q6_Kssm_outweight (12,902,400 B) —the
ssm_outprojection, taken by a HIP add-on kernel rather than byggml_mul_mat_f32_f32_wmma.activation_format=9commands in that program: 0. Noweight_format=44/46matmul command either.So the defect as reported (fused gate -> Loom packed-wmma with
activation_format=Q8_1) is reachable by a graph I canwrite, but not by this model's graph: the model's GDN layer swallows the projection into the GDN dispatch, and the Q8_1
buffer is handed to a different consumer that has its own (HIP) parameterisation.
Two caveats, so nobody over-reads this:
program, because the run was killed during the weight load (~90 s in) before any request was served. The epilogue fusion
is what decides, and it was present in the shape I captured; I have not verified it for the decode shape.
hip:k_f69572a806c8handles the Q8_1 x4 activation correctly is a separate, still-open question, and it is theone that actually matters for output quality. Nothing here answers it — the run produced no completions.
Incidental but useful for anyone chasing the same class of bug: the Q8_1 x4 hand-off itself is laid out consistently in this
model. The producer writes 96 rows of 4 blocks (144 B each,
d/sin an 8xf16 header) and the consumer reads 2 tokens x192 blocks, so group g covers activation elements 128g..128g+127 = head row g; the byte counts and the grouping agree
(96 x 144 = 2 x 192 x 36 = 13,824). So on this path the layout is not the suspect — the Loom consumer's read is.
Separately, and on my own account: that load reached 109.21 GiB of GTT before my GTT cap killed it (0 gfxhub faults, no
MODE2 reset, box healthy afterwards). The 76 GiB model does not fit next to this box's other tenants even at
-c 2048, whichis why no completion ran. I will not re-attempt it on a shared box.
bong-water-water-bong commentedon Nov 4, 2026
Important correction, and it cuts against the severity of this issue: the reproducer does not reproduce in a real model, at the
model's exact command shape. A real Qwen3-4B-Q4_K_M decode program contains this very pair, and that model on this binary is
numerically correct to f16 level.
The model-level quality measurement (first one this lane has taken)
~/lb/aval/tattn-ppl-ab/20261104T070316Z, same binary, same GGUF, same corpus,llama-perplexity -c 512 --chunks 2:-ngl 99)-ngl 0)0.04 % apart — that is f16 noise, not a corrupted projection. Greedy text agrees too: two prompts, HRX vs CPU, identical for the
first ~46 tokens and diverging after that (the normal near-tie drift of greedy decoding, not garbage from token 1).
And the pair really is in that model's plan
Dumped from Qwen3-4B-Q4_K_M with
GGML_HRX_DUMP_COMMAND_PROGRAM_DIR(~/lb/aval/tattn-reach-qwen34b/20261104T063929Z), programprogram-2-shape-a36a80b886d50fdb(185 commands, 1-token decode shape) contains exactly:4096 -> 2560with 1 token is the model's attention output projection (o_proj). If that node were producing NaN in ~half itsrows, the perplexity above would be nonsense. It is not. So either that program is not what executes for this model's decode, or
the kernel behaves differently in the model's environment than in my harness.
My harness fails at the model's exact command shape
~/lb/aval/attn-kquant-ksweep/20261104T065400Z(same arm as the issue, sweepingKatN=2560,token_count=1, Q4_K, viacommon.rmsnorm_binary_q8_1_x4->common.mul_mat.f32_f32_decode):Every K fails here, including 4096 — the same size that is fine in the model. So the shape is not the discriminator either, which
means the difference is in the environment of the compute, not in the dispatch choice.
What I think the remaining difference is (hypothesis, untested)
The probe computes directly on
HRX0(ggml_backend_alloc_ctx_tensors+ggml_backend_graph_compute), with a tiny transientarena (3 transients, 31,488 B in the earlier dump). The model runs the same command through the llama.cpp scheduler and residency
path. The symptom that fits both worlds is a read past the end of the Q8_1 activation binding: in the model the bytes after the
4,608-byte activation are other live, finite tensors (so the model degrades by a small, hard-to-see amount rather than exploding);
in my probe they are the tail of a small arena and then unmapped memory (NaN). That is consistent with:
It is not established — I have not measured the read extent. The test I would run next, and did not have time for tonight, is to
put the same graph through
ggml_backend_sched(CPU + HRX0) instead of direct compute, and/or pad the arena, to see whether thefailure follows the allocator rather than the graph.
What this issue is, for now
A real, deterministic, model-free reproducer of something in the Q8_1-activation + packed-quantised-weight read. Not evidence
that a shipped model is broken: the one model I could measure with this pair in its plan is at perplexity parity with CPU, and the
76 GiB model I would have liked to measure cannot be loaded on a shared box. Treat it as latent until the direct-compute vs
scheduler discrepancy above is resolved; I will keep the harness and the dumps in place and update here.
bong-water-water-bong commentedon Nov 4, 2026
Addendum with the last piece of evidence, then I am leaving this where it is until the discrepancy is explained.
Does the decode of that model actually run the program containing the pair? The dump gives the plan; the host-gap profiler
gives what ran. One more Qwen3-4B run with both on (
~/lb/aval/tattn-decode-prog/20261104T070728Z, two greedy 64-tokencompletions):
hrx host gap TOTAL: graphs=130 wall=2119.411 ms build=8.978 ms launch=1719.963 ms dispatches=25916→ 199 dispatches pergraph on average.
activation_format=9+weight_format=44); the larger ones are the prefill shapes.against CPU's 13.3347.
That closes the loop as far as this issue should carry it: a deterministic model-free reproducer of something in the
Q8_1-activation + packed-weight read, at a shape and dispatch that a real model executes without any visible numerical damage. The
remaining difference is in how my harness sets the compute up, not in what the model runs.
What I have excluded on that difference (all measured, harnesses in
~/wt/hrx-probe/kquant_probe.cppand~/lb/tattn-kquant-*.sh): dispatch choice, weight type (Q4_K/Q6_K), producer (two different Q8_1 writers), input size (swept2048-8192, all fail in the harness including the model's own 4096), row tiling, and "the packed words are never materialised"
(
prepare_command_programalways callsmaterialize_host_bindings, and no failure status was logged).Highest-value next test, if someone picks this up:
materialize_host_bindingsprefersbinding.host_dataand falls back tobinding.bufferfor the transformation source. A llama.cpp model registers weights with a host pointer; my harness only has thedevice buffer. A bug on the device-buffer source path would produce exactly this asymmetry — garbage in a harness, parity in a
model, and zeros when the weights are zeros. I did not get to test it tonight.
Closing my end: no fix proposed, no model-quality claim made, reproducer and dumps left in place.
bong-water-water-bong commentedon Nov 4, 2026
RETRACTION: this issue as filed is my harness bug, not an engine defect. The 2x2 that settles it, and what is left (a small silent-garbage guard)
Summary up front: the divergence depends on one flag — whether the weight tensor's buffer is flagged
GGML_BACKEND_BUFFER_USAGE_WEIGHTS. My reproducer allocated every tensor withggml_backend_alloc_ctx_tensors(one buffer, usage
ANY), sobinding.weightwas false,materialize_host_bindings(
runtime/command-program-executor.cpp:581) never requested the repack, and the kernel — whose packed layout had alreadybeen chosen from the tensor type by the dispatch builder — read raw unpacked Q6_K/Q4_K bytes as packed rows. The model
flags its weight buffers, which is why it is at CPU parity while the probe was garbage.
The 2x2 (same binary, same arm
ngated, same leaves, same weights; only the weight binding changes)ANY79497e64…WEIGHTSsource.device_buffer(download)2598052d…WEIGHTSsource.host_data2598052d…ANY79497e64…Rows 2 and 3 are byte-identical to each other, and rows 1 and 4 are byte-identical, so the materialization source is
not the variable at all — the device-source download path is correct — and the usage flag is the whole story. Q4_K
reproduces it the same way (
weight_format=44): usageWEIGHTS→ 1.0e-02, usageANY→ same 1920/2560 garbage. Repeatruns of each arm are byte-identical, so this is deterministic in both directions, not noise.
What that means for the earlier comments on this issue
Every "the fused
rmsnorm_gate→ packed-quantised matmul pair returns garbage" row I posted above was produced in theusage-
ANYconfiguration, i.e. it was measuring my harness, not the engine. Same for the stage bisect (KP_STAGE), theweight-type split (Q4_K/Q6_K), the row-map and distinct-rows arms, and the zero-weight control. They are not evidence of a
defect and should not be read as such. My apologies — the check I was missing is one line of
ggml_backend_buffer_get_usageon the weight buffer before comparing any two arms.
What survives, and it is a different (smaller) thing
The engine is correct whenever the packed-layout binding is a weight binding, on either materialization source. The
remaining gap is that the layout choice and the repack are gated by different things:
build_mul_mat_dispatchpicks arepacked weight layout from the tensor's type, while
materialize_host_bindingsonly repacks when the buffer carries theWEIGHTSflag. A quantisedsrc0in a buffer without that flag therefore runs the packed-layout kernel over unpackedbytes and returns silently wrong numbers. Nothing in a normal model does that (which is why perplexity is at parity), but
the failure mode is silent.
PR opened for the bounded guard — refuse the command program with one line naming the value, the layout and the missing
flag, instead of running it: #192. Measured there: the two
ANYconfigurations above now fail loudly, the twoWEIGHTSconfigurations are unchanged at 9.7e-3, Qwen3-4B-Q4_K_M perplexity on HRX0 is 13.3298 +/- 2.24917, identical to four
decimals and with zero refusals logged, and
test-backend-ops -b HRX0passes MUL_MAT, MUL_MAT_ID, GET_ROWS and CPY 2/2backends with no refusal.
I am leaving this issue closed as filed-in-error rather than repurposing it: the guard has its own PR and does not need an
issue that carries four wrong comments above it. If you would rather keep it open as the tracking entry for the guard,
reopen and I will move the content there.