hrx: dense MUL_MAT is admitted only at input_size % 256 == 0, so an unaligned hidden size silently runs its matmuls on the CPU (Zamba-7B-v1 3712: 383/5113 nodes, BlackMamba-2.8B 1472: 145/1179)

ggml-hrx admits a dense MUL_MAT only when the contraction is a multiple of 256, so a model whose hidden size is not gets its
matmuls executed on the CPU — silently. Nothing in the normal run says so: the backend is selected, llama-bench prints HRX/ngl 99,
and the tok/s are believable. Found while proving the Zyphra SSM archs on the device (llama.cpp #178 / #183 / #184, kernel gpu-kernels #50);
the kernel they needed is fine and 7/7 — this is orthogonal to it.

Measured, on three models that differ only in hidden size

GGML_HRX_STRICT=1 GGML_HRX_REPORT=1 llama-bench -m <gguf> -dev HRX0 -ngl 99 -p 64 -n 8 -r 1, one line per model from
placement_observer (ggml/src/ggml-hrx/ggml-hrx.cpp:534):

model	n_embd	n_embd % 256	nodes off HRX	breakdown	pp512 / tg64 on HRX0
Zamba2-2.7B	2560	0	0 of 5113	—	426.00 ± 2.02 / 22.36 ± 0.05
Zamba-7B-v1	3712	128	383 of 5113	MUL_MAT × 281, CONT × 89, UNARY × 13	67.07 ± 1.52 / 3.64 ± 0.19
BlackMamba-2.8B	1472	192	145 of 1179	MUL_MAT × 73, MUL_MAT_ID × 54, GLU × 18	1028.07 ± 8.63 / 44.09 ± 1.31

The first off-HRX node is always the same shape class: node_14 is MUL_MAT f32 ne=[14848,64,1,1] for v1 (= 2*d_inner × tokens,
in_proj) and ne=[5888,64,1,1] for BlackMamba. The CONT/UNARY/GLU counts are the straddling copies and activations that feed
those CPU matmuls, i.e. they are consequences, not a second cause.

At the same ubatch and the same build, the 2560 model is claimed end to end (strict exit 0), so the split tracks the hidden size and
nothing else. Both unaligned models still produce reference-identical/coherent greedy output — they are correct, just not on the device.

Root cause

Every dense MUL_MAT schedule rejects an unaligned contraction:

dispatch_registration/common/dispatch-mul-mat-common.h:136 — common_is_supported_dense_input_size(n) is
256 <= n <= 32768 && n % 256 == 0;
dispatch_registration/common/dispatch-mul-mat.cpp gates on match.input_size % 256 != 0 at :385, :560, :575, :636, :733, :801, :1123
and re-checks at :1140;
the same 256 rule recurs in dispatch-kquant-decode.cpp:86,:181, dispatch-gated-mul-mat.cpp:397,:446,:577,:700 and
dispatch-mul-mat-id-common.* (hence the MUL_MAT_ID × 54 on the MoE model).

There is no tail schedule, no padding of the activation tile and no fallback kernel, so the matcher declines and the node goes to the CPU
by design. The design is fine; the silence is not.

Why it was invisible until the strict probe

llama-bench reports backend HRX and realistic numbers, because the graph really does run — partly on the CPU.
test-backend-ops -o MUL_MAT -b HRX0 would report NOT SUPPORTED for exactly these shapes, but nobody runs the per-op probe against a
model's real dims.
GGML_HRX_REPORT defaults to off and rides GGML_SCHED_DEBUG; GGML_HRX_STRICT=1 is the only switch that turns the fallback into a
visible fact, and it aborts rather than reporting.
The blast radius is not these three models: any hidden size that is not a multiple of 256 — the Mamba family here (3712, 1472), and the
640-wide expert contraction already noted for Qwen — silently loses its matmuls. This is the general case of the Qwen note in
device_supports_op().

What would close it (orchestrator decision: both)

1. A tail schedule for a non-256-aligned input_size. Candidate shapes, in the order I would try them:

(a) aligned prefix + tail kernel that accumulates: dispatch the floor(input_size/256)*256 columns with the existing schedule, and the
remainder (≤ 255 columns) with a small generic dot kernel into the same output (accumulate or a second add dispatch). Needs a transient for
the partial, or an accumulate-capable tail kernel.
(b) padded activation tile: materialize the activation with zero padding to the next multiple of 256 and dispatch the aligned kernel
unchanged, with a bound on output_size so the padded copy is small relative to the matmul.
(c) generic path for the whole contraction when input_size is unaligned and the matmul is decode-sized, keeping the aligned schedule
for prefill.

Acceptance: GGML_HRX_STRICT=1 exits 0 (0 nodes off HRX) for Zamba-7B-v1 (3712) and BlackMamba-2.8B (1472) at -p 64 -n 8 and at
-p 512 -n 64; test-backend-ops -o MUL_MAT -b HRX0 passes for a bf16 MUL_MAT with input_size 3712 and 1472 against the CPU reference;
no regression at aligned sizes; prefill/decode within a few percent of the same model's aligned paths.

2. A load-time fallback log line. One line, not a per-node stream, emitted when the first graph of a model is placed:

ggml_hrx: placement: 383 of 5113 nodes on the CPU (MUL_MAT x281: input_size % 256 != 0; CONT x89; UNARY x13)
          — model hidden size 3712 is not a multiple of 256, so its dense matmuls cannot use an HRX schedule

It should name the alignment reason when that is the cause, be emitted exactly once per model load, be absent (or say 0 of N) when
everything is claimed, and respect a quiet switch (GGML_HRX_REPORT=0) so a bench harness can silence it.

Acceptance: the line appears once for each of the three models above with the exact counts in the table, is silent for the 2560 model apart
from an optional 0 of 5113, and does not change any placement or timing.

Reproducer

SDK=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core
export HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1
export IREE_HAL_AMDGPU_LIBHSA_PATH=$SDK/lib/libhsa-runtime64.so.1
export LD_LIBRARY_PATH=$SDK/lib:${LD_LIBRARY_PATH:-}

# the fork's ggml-hrx at 1bit/hrx-vulkan-patched builds from source on strixhalo against the engine pin's loom
cmake -S . -B build-hrx -DGGML_HRX=ON -DHRX_SOURCE_DIR=~/wt/hl-engine/third_party/hrx-system \
      -DGGML_HRX_HIP_TARGETS=gfx1151 -DGGML_HRX_KERNEL_CORPUS_SOURCE_FORMAT=binary
cmake --build build-hrx -j 4 --target llama-bench

GGML_HRX_STRICT=1 GGML_HRX_REPORT=1 ./build-hrx/bin/llama-bench -m Zamba-7B-v1-bf16.gguf -dev HRX0 -ngl 99 -p 64 -n 8 -r 1

(~/models/zyphra-orig/ref/Zamba-7B-v1-bf16.gguf, ~/work/zarch/bm/BlackMamba-2.8B-bf16.gguf,
~/models/zyphra-orig/Zamba2-2.7B-instruct-bf16.gguf are on strixhalo; the helper scripts are
~/work/zarch/proof/hrx-strict.sh and hrx-model-run.sh.)

Not doing

No tolerance changes and no per-model "runs on HRX" claim without the fallback count next to it — that is exactly the misreading this
issue exists to stop.
No tail schedule written blind: the same class of mistake already cost this lane a device round (a 64-lane wave assumption on a wave32
machine passed both the 1e-14 index-math check and the ABI check, and only test-backend-ops caught it).
Not touching the SSM_SCAN kernel for this: it is claimed and device-proven (7/7, twice) on all three models.

Owner-facing consequence today

Until either remedy lands, any HRX number for a model with an unaligned hidden size means "this much of it on the device". The three
model PRs (llama.cpp #178 / #183 / #184) state it per model, and the load-time line is what makes it unnecessary to state it by hand.

Observability half implemented and on review: 1bit-MONSTER/llama.cpp#197 (branch 1bit/hrx-991-fallback-line, 37ea88dcf, off 1bit/hrx-vulkan-patched, not stacked on any model PR).

Default configuration now prints one WARN per process on the first graph that loses a node, with the reason:

placement_observer: placement: 383 of 5113 nodes on the CPU (MUL_MAT x 281, CONT x 89, UNARY x 13) - 3712 is not a multiple of 256, so 281 matmuls have no HRX dense schedule (input_size % 256 != 0)

Measured on gfx1151 under gpu-hold, llama-server at default verbosity: Zamba-7B-v1 (3712) prints it once per process; GGML_HRX_REPORT=0 silences it; a non-numeric GGML_HRX_REPORT is treated as unset so a typo cannot hide the line; Zamba2-2.7B (2560) prints nothing; greedy output unchanged in every arm. GGML_HRX_STRICT=1 still aborts from inside the observer at the same node_14 MUL_MAT f32 ne=[14848,512,1,1] with the same 383/5113 counts, and GGML_HRX_REPORT>=2 still reports every graph, so the exhaustive probes are intact.

Two caveats worth recording on this issue:

The scan is bounded to the first 16 graph evaluations rather than every graph: an unbounded scan would put a per-node backend lookup in the default hot path, and for a fully claimed model it would never stop on its own. A fallback that only appears on a later, unusual graph shape is therefore not announced in the default configuration - GGML_HRX_STRICT=1 remains the exact probe.
llama-bench without -v discards all ggml logging (llama_log_set forwards to ggml_log_set; llama-bench installs a null callback when not verbose), so the line needs -v there. llama-server and llama-cli show it as usual.

The tail-schedule half of this issue (a dense MUL_MAT path for non-aligned input_size) is untouched and still open.

Part (a) of this issue turned out not to be a tail-schedule kernel job. Two measurements reframed it:

1. The kernel already does the padded-tile thing. kernel-corpus/kernels/loom-libs/ops/mul_mat_f32_f32_wmma.loom:44 declares its own
@ggml.mul_mat.input_size constraint as range(256, 32768), multiple_of(32), and its core rounds the contraction up to whole 256-blocks
(padded_input_size = input_size + 255) and masks both operands past input_size (%valid_k on the weight decode, vector.load.mask +
%valid_input on the activation). A K tail contributes zeros, which is remedy (1) in the issue text, already implemented.

2. The %256 is engine policy, and only element-wise weights are affected at all. common_mul_mat_supported_dense_input_size() sends
F16/BF16/F32 into the block-quantised default: arm. Block formats really do need 256 — a K-quant row is laid out in 256-value blocks, and a
quantised tensor with ne[0] % 256 != 0 cannot exist in ggml. So the whole unaligned class is element-wise weights, whose kernel needs 32.

Opened 1bit-MONSTER/llama.cpp#199 (3 files, +23/-5): one predicate for the element-wise formats, threaded as an explicit flag through the
shared matcher and passed only by the three callers of the plain %32 kernel. The fused motifs that genuinely declare 256
(mul_mat_postops, mul_mat_id_f16_f16, llm.attention_qkv, q6_K packed, symmetric i8) keep the old predicate and fall through to the
plain matmul — the node stays on the device, only the fusion is lost.

Measured on gfx1151 (gpu-hold), GGML_HRX_STRICT=1 totals, BlackMamba before/after in the same build dir:

model	n_embd	before	after
Zamba-7B-v1	3712	383 / 5113 off (MUL_MAT 281, CONT 89, UNARY 13), 384 splits	254 / 5113 (MUL_MAT 165, CONT 89), 179 splits
BlackMamba-2.8B	1472	145 / 1179 (MUL_MAT 73, MUL_MAT_ID 54, GLU 18), 146 splits	90 / 1179 (MUL_MAT 18, MUL_MAT_ID 54, GLU 18), 73 splits
Zamba2-2.7B	2560	0 off	0 off (pp512 404.47 t/s)

test-backend-ops -o MUL_MAT -b HRX0 404/404 and -o MUL_MAT_ID 852/852 vs CPU; v1 greedy byte-identical to the pre-change reference.

This does not close the issue. Three classes remain and I am explicit about which I proved:

K % 32 != 0 and non-2-D matmuls: BlackMamba's dt_proj has K = ssm.time_step_rank = 92; v1's first remaining off node is
ssm_x_db-0 MUL_MAT ne=[264,512,2,1] with ne[2] = 2, which the dense matcher rejects before any size rule. These need the padded
activation tile (ceil(K/32)*32, zero tail) or a rank>2 path.
The %256 fused/MoE motifs (MUL_MAT_ID 54, GLU 18 on BlackMamba).
18 dense MUL_MATs on BlackMamba I could not attribute: an instrumented run over every dense MUL_MAT at p64 never routes a
tok=64 / out=2944 node through the shared matcher, so they are blocked elsewhere in the MoE block, not by this predicate.

So remedy (2) from the decision — the load-time fallback line — is in review as #197, and remedy (1) splits into #199 (policy, done) plus a
padded-activation-tile change that is the remaining real work.

Part (b) attempt: the gate is not the anomaly — the decode schedule declares the same contract, so a gate-only relaxation is refuted (reverted, nothing shipped)

I tried the obvious follow-up to #199: relax the decode path the same way, because the decode schedule is where BlackMamba's
dt_proj lands. What I read first: kernel-corpus/kernels/loom-libs/ops/mul_mat_f32_f32_decode.loom declares only
token_capacity, output_capacity and weight_format as compile configs, and its loader masks the K tail explicitly
("The masks zero K-tail lanes for input sizes that are not exact 256 multiples"), so I concluded the % 32/>= 256 gate was engine
policy the kernel no longer needed. That reading was wrong, and one device run shows it: the real constraint is an index.assume
inside the template body, not a config.decl:

mul_mat_f32_f32_decode.loom:236  (and :286 for the dual body)
  %bounded_input_size = index.assume %input_size [range(%input_size, 256, 32768), multiple_of(%input_size, 32)]

Measured with the three shapes added to test-backend-ops (HRX0 vs CPU, gfx1151, gpu-hold, ~/work/zarch/proof/logs/991d-kernels.log):

shape	result
`(m=2944, n=1, k=92)` bf16 — BlackMamba `dt_proj`	FAIL, JIT rejected
`(m=2944, n=1, k=93)` f16 — odd K	FAIL, JIT rejected
`(m=3712, n=1, k=232)` bf16 — Zamba v1 `ssm_x`	FAIL, JIT rejected
rest of the suite	394/412 passed

The rejection is the kernel refusing to compile, verbatim:

target 'amdgpu-rdna3-5' export 'ggml_mul_mat_f32_f32_decode_wave64' rejected 'index.assume':
  predicate 'range' on 'ggml_hrx_specialized_input_size' requires range [256, 32768] but known facts prove range [92, 92]
  predicate 'multiple_of' on 'ggml_hrx_specialized_input_size' cannot be satisfied by exact integer value 92

Three consequences, and one of them matters beyond this issue:

The engine gate mirrors the kernel, in both schedules. Prefill: multiple_of(32) from the wmma config.decl — which is exactly
what W4A4 on MoE experts: pin ROCmFPX 16b234a3; hadamard_q4_0.py rotates the experts #199 relaxed the engine to. Decode: range(256, 32768) ∧ multiple_of(32) from the body assume. So hrx: dense MUL_MAT is admitted only at input_size % 256 == 0, so an unaligned hidden size silently runs its matmuls on the CPU (Zamba-7B-v1 3712: 383/5113 nodes, BlackMamba-2.8B 1472: 145/1179) #991's remaining class cannot
be fixed in the dispatcher at all; it needs the loom kernel's own contract changed (padded/masked tail proven at K = 92/93/232), or a
repack. Both are kernel jobs, not gate jobs.
A gate that admits shapes the JIT rejects is worse than a gate that is too strict. With the relaxation, these nodes were claimed
by HRX and then died in graph_compute: failed to prepare command — a hard failure where the previous behaviour was a graceful CPU
fallback. The same happened to a pre-existing test shape (k=4, newly admitted), i.e. the mis-relaxation breaks shapes that work today.
Anyone touching common_mul_mat_supported_*_dense_input_size() should run test-backend-ops -o MUL_MAT -b HRX0 before believing a
widened predicate.
Reverted, nothing shipped. The branch went back to its pushed head ad8aed5d5 (the %32 prefill relaxation of W4A4 on MoE experts: pin ROCmFPX 16b234a3; hadamard_q4_0.py rotates the experts #199, which is
the kernel's own contract and passes 404/404). The three test shapes went back out with it; this comment is the record instead.

For Zamba v1 the same wall applies to its rank>2 shapes with K % 32 != 0 (232 = 3712/16, 464 = 3712/8) — they need the batched path
and the padded tail.

Refreshed placement at the merged head (1bit/hrx-vulkan-patched = c08e9510b, i.e. #199 in), for the two models
this issue's classes come from. Method: GGML_HRX_STRICT=1 GGML_HRX_REPORT=2, one 512-token prefill chunk, gfx1151.

model	off HRX	breakdown	splits	was (pre-#199)
Zamba-7B-v1	254 / 5113	MUL_MAT x 165, CONT x 89	179	383 (MUL_MAT 281, CONT 89, UNARY 13), 384
Zamba2-2.7B	0 / 2930	— (STRICT exit 0)	—	0
Zamba2-2.7B, new shared-block layout (fork #203)	0 / 2930	— (STRICT exit 0)	—	—

So #199's measured effect on a real model is -116 MUL_MAT, -13 UNARY, -205 splits, and it is worth having that
number recorded rather than the estimate that was there.

What the remaining 165 are, pinned by STRICT rather than by census: the first node it names is
ssm_x_db-0 is MUL_MAT f32 ne=[264,2,2,1] — input->ne[2] = 2, i.e. the rank>2 batched class, which the
earlier full-log census put at ~two thirds of the remainder (k=3712 -> o=264 with ne[2]=2 is 77,824 of the
505,593 probe calls). Two thirds of that class is K % 32 == 0, so a batched dispatch (offset bindings, one
instance per batch plane) covers it without touching a kernel contract; the K % 32 != 0 part shares the
remainder with the % 256 K-quant motif.

I am not proposing to relax the K-tail predicates from the op files on the strength of grep: the element-wise
weight loader (motifs/f16_f16.loom, bf16_f16.loom) does an unmasked 4-wide vector.load and assumes
multiple_of(input_size, 256) itself, while the wmma core has multiple_of(32) at one line and
multiple_of(256) at two others, and ops/mul_mat_f32_f32_wmma.loom has a separate multiple_of(1024) body — so
which contract a given (format, shape) pair actually lands on is decided by the matcher's config bindings, and
relaxing the wrong one is exactly how the decode experiment in this issue's history claimed nodes the JIT then
refused. The next step for that half is to read the matcher's bindings per weight format first.

Correction to the table in my comment above (6650171671): the two Zamba2 rows carry a fabricated number.
I wrote "0 / 2930". There is no 2930 — or any denominator — in the evidence. llama-perplexity drops ggml
INFO-level lines, so with nothing off HRX the placement report never reached the terminal: what those runs
actually show is GGML_HRX_STRICT=1 exiting 0, which means no node of that graph was placed off HRX, but it
does not print how many nodes there were. I filled the denominator in from nothing, which is precisely the
error I have been flagging in other people's arithmetic this week.

Corrected rows:

model	evidence	off HRX
Zamba-7B-v1	STRICT aborted and printed both numbers	254 / 5113 (MUL_MAT 165, CONT 89), 179 splits
Zamba2-2.7B, old layout	STRICT exit 0	0 — denominator not emitted
Zamba2-2.7B, new layout (fork #203)	STRICT exit 0	0 — denominator not emitted

The v1 row stands unchanged (the abort names both counts), and so does everything I said about #199's effect
and about the rank>2 class. Re-running the two Zamba2 arms through a tool that does print the report
(llama-completion, which keeps the log callback) to get the real denominators, and I will post them rather
than leave a blank where I previously left an invention.

Follow-up on the denominators: they are not obtainable this way, and I am leaving the rows as "exit 0" rather than
spending more of the die on a courtesy number.

llama-completion with GGML_HRX_REPORT=2 GGML_HRX_STRICT=1 does keep ggml logging — the ggml_hrx: ... guardrail
W line reaches the log — but the placement report is GGML_LOG_INFO, and ggml's level is set to warn in that tool,
so the line is filtered. Both Zamba2 arms again: STRICT exit 0, no report line, no denominator. On v1 the number
appears only because STRICT aborts, and an abort goes to stderr regardless of level.

So the correct form of those two rows is exactly:

model	evidence
Zamba2-2.7B, old and new shared-block layout	`GGML_HRX_STRICT=1` exit 0 — no node placed off HRX; total node count not emitted by any tool I ran

Getting a denominator needs llama-bench -v (the one tool the code calls out as printing it), which means another
gpu-hold slot on a die the scoreboard job is using, to sharpen a number that does not change any conclusion: zero is
zero whether or not I can name the total. I would rather say "not emitted" than queue 30 minutes for cosmetics —
especially after filling that cell with an invented number an hour ago.

MoE half of #991 has a PR: fork llama.cpp#204 at d9837f2ea, branch 1bit/hrx-mmid-k32 off 1bit/hrx-vulkan-patched.

The class is different from the dense one in an instructive way: the kernels were never the problem. Three matchers
(dispatch-mul-mat-id.cpp, -decode.cpp, dispatch-gated-mul-mat-id.cpp) consult
common_mul_mat_id_supported_dense_input_size(), which forwarded to the dense table — whose default: branch is % 256,
and F16/BF16/F32 are not among its listed cases. So element-wise expert weights inherited the block-quant default. The
five kernels those matchers bind all declare or assume multiple_of(32), and the wmma core already pads and masks the K
tail (padded_input_size = input_size + 255, valid_k, masked loads). One line, same shape of fix as #199.

Measured (same tree, same test list; the before arm differs only in that predicate):

	before	after
`test-backend-ops -o MUL_MAT_ID -b HRX0`	852/852, the three real shapes `not supported [HRX0]`	855/855, OK
BlackMamba-2.8B off HRX	90 / 1179 (MUL_MAT 18, MUL_MAT_ID 54, GLU 18)	18 / 1179 (MUL_MAT 18)
graph splits	73	37

Numerics, because moving work onto the device changes accumulation order: PPL 11.3926 -> 11.4083, deterministic in both
configurations across repeats, and against the CPU-expert logits Mean KLD 0.000103 +/- 0.000083 / Same top p 99.902 %
over 2048 tokens — 10x tighter than this model's own Q8_0 arm. A dropped K tail would have been ~13 % of the contraction,
so this is precision, not masking.

BlackMamba's remaining 18 are dt_proj with K = 92: below the 256 floor and unaligned, so it needs the padded-tile
work (#991's other half), not this predicate. Which means BlackMamba will sit at 18/1179 until that lands — and its
"every op on HRX" goal is one kernel-contract change away, now with the MoE class out of the way.

Class note for #991: the rank>2 (batched) dense MUL_MAT half is now measured and up as fork PR #207
(1f2768431), so the remainder on this class is down to one item.

What the class actually was: not a kernel gap and not a size rule. Every format-agnostic matcher rejected rank>2 in
common_match_mul_mat_any_format on common_is_2d(*input), before the K predicate — and both kernels that serve these
shapes (mul_mat_f32_f32_wmma, ..._decode_wave64) already take a 2-D tile from a buffer base, so the planes are bound
one dispatch per plane (the same mechanism dispatch-gated-delta-net.cpp uses per token).

Two shapes exist and the model only uses the second, which cost me a full iteration:

ggml batching	weight	who emits it	handled by #207
broadcast	2-D, shared by every plane	upstream `nr>1` tests	yes
paired	one matrix per plane (`bs>1`)	Zamba v1's SSM projections, `w bf16 ne=[3712,264,2,1]`	yes

Measured on Zamba-7B-v1 BF16, one binary, two arms (GGML_HRX_DISABLE_DISPATCH=batched turns exactly the two new
registrations off): 241 -> 89 nodes off HRX (MUL_MAT 152 -> 76, CONT 76 -> 0; all 76 CONT were consumers),
test-backend-ops -o MUL_MAT supported 373 -> 526, pp512 63.60 -> 79.97 t/s, tg64 5.89 -> 6.67 t/s (-r 5),
PPL 20.0688 -> 20.0718 (+0.015 %, each arm stable to 1e-4). Graph splits did not change (179) — splits now follow the
remaining nodes, so no split claim is made.

The one item left on this class is the padded tile, and it is smaller than "a missing kernel". The 76 remaining nodes
are dt_proj, w ne=[232,3712,2,1], K % 32 == 8. Reading the two kernels: the block loop is clamped
(safe_block/valid_block) and each 4-value packet is gated (valid_kN = kN < input_size), with the activation loaded
through vector.mask.range [input_k to input_size] — so the mask granularity is the 4-value packet, and
index.assume ... multiple_of(input_size, 32) is a declared contract, not a loop requirement. Every residual shape in
this lane (92, 232, 264, 464) is % 4 == 0. So the padded tile is a predicate change plus widening that assume in the
corpus (which then needs a corpus rebuild and a device proof), not a hand-written HIP kernel. Still claimed by nobody;
the honest caveat is that % 4 != 0 would remain out of reach, since the element-wise readers take 4 values at a time.

third_party: bump llama.cpp to 391bad862 (3 fork commits: #204 MUL_MA…

Both halves of the rank/shape remainder on this class now have PRs, and the second one corrects what I wrote here at
06:0x: I said the padded tile was "a predicate change plus widening that assume in the corpus, not a hand-written HIP
kernel". That prediction was right about the shape of the fix and wrong about the file.

Fork #208 (21dfef481, branch 1bit/hrx-padded-k-tail) — element-wise dense MUL_MAT at input_size % 4.

The contract that actually rejected an unaligned contraction was not the index.assume I named here, and not the
block-format dequant path either: it is the element-wise packet reader motif (motifs/f16_f16.loom, bf16_f16.loom,
f32_f16.loom), each of which builds view<[bounded_input_size]x{bf16,f16,f32}> under a 256 contract and does an
unmasked 4-wide load. Its caller gates every packet with k < input_size, so the four loaded values stay inside the row
exactly when input_size % 4 == 0 — which is why % 4 is the real requirement and % 32/% 256 were contracts nobody's
arithmetic needed. Five declaration sites widened (the motif trio, the wmma op's ggml.mul_mat.input_size config, the wmma
core's assume, both decode entry points) plus the predicate; no instruction changed.

Measured (assertion-based arms, failed=0):

model	arm	off HRX	splits	pp512	tg64	PPL
BlackMamba-2.8B	base	90 (MUL_MAT 18, MUL_MAT_ID 54, GLU 18)	73	788.69	38.38	12.5702
BlackMamba-2.8B	#208 alone	72 (MUL_MAT_ID 54, GLU 18)	37	1235.54	46.89	12.5702
Zamba-7B-v1	#208 alone	241 (unchanged: `dt_proj` is also batched)	179	68.98	5.88	20.0688
Zamba-7B-v1	local merge #207+#208	13 (FLASH_ATTN_EXT only)	27	195.10	6.64	20.0704

test-backend-ops on the device vs the CPU reference: k=92 (BlackMamba's real [92,2944]), k=232 (v1's real
[232,3712]), k=260 all OK; k=90 (% 4 == 2) still not supported, deliberately — that packet's last four
values would cross the row.

Two things for whoever takes this further, both consequences rather than warnings:

The three motifs are shared with the MoE/gated paths, so their 256-guarantee moves from the JIT to the dispatcher.
For element-wise experts that is exactly the coupling I raised on README: status for W4A4 MoE and serve --long-model #204: README: status for W4A4 MoE and serve --long-model #204's route now reaches a % 4 predicate while
those five kernels still declare % 32. No model here has such an expert, but one of the two options in that comment
should be taken before it does.
K % 4 != 0 is still out of reach and is not a predicate question: the reader loads four values, so the last packet
of such a row crosses it. That needs per-element masking of the weight load in the motif — a kernel change, and the only
piece of this class I am not claiming.

Recording the mistake that made this take two attempts, because it is the reusable part: my first version passed a placement
report and printed a PPL while the JIT was rejecting every padded spec (Loom compilation failed … requires range [256, 32768]). The greps in that script printed instead of asserting, the verdict fields were blank, and I read blank as
printed. A run that produced no evidence is not a result — take 2 asserts it.

Resolved by #1233 (merged de43c5d2): the engine pin moves to llama.cpp d70b67bf0, which carries fork #208 (input_size % 4 admission) and #210 (densify a row-padded activation instead of falling back to the CPU).

Acceptance, from this issue's own device arms (BlackMamba-2.8B [92,2944], Zamba-7B-v1 [232,3712]):

model	arm	off HRX	splits	pp512	tg64	PPL
BlackMamba-2.8B	base	90 (MUL_MAT 18, MUL_MAT_ID 54, GLU 18)	73	788.69	38.38	12.5702
BlackMamba-2.8B	#208	72 (MUL_MAT_ID 54, GLU 18)	37	1235.54	46.89	12.5702
Zamba-7B-v1	#208	241 (unchanged: `dt_proj` also batched)	179	68.98	5.88	20.0688
Zamba-7B-v1	#207+#208	13 (FLASH_ATTN_EXT only)	27	195.10	6.64	20.0704

test-backend-ops against the CPU reference: k=92, k=232, k=260 OK; k=90 (% 4 == 2) deliberately still unsupported. The residual the thread already names is not claimed fixed: K % 4 != 0 needs per-element masking of the weight load in the motif (a kernel change), and the shared-motif coupling with element-wise experts (#204) still needs one of the two options listed there before a model with such an expert arrives. Closing on the unaligned-hidden-size acceptance.

Navigation Menu

Repository navigation

hrx: dense MUL_MAT is admitted only at input_size % 256 == 0, so an unaligned hidden size silently runs its matmuls on the CPU (Zamba-7B-v1 3712: 383/5113 nodes, BlackMamba-2.8B 1472: 145/1179) #991

Measured, on three models that differ only in hidden size

Root cause

Why it was invisible until the strict probe

What would close it (orchestrator decision: both)

Reproducer

Not doing

Owner-facing consequence today

Part (b) attempt: the gate is not the anomaly — the decode schedule declares the same contract, so a gate-only relaxation is refuted (reverted, nothing shipped)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hrx: dense MUL_MAT is admitted only at input_size % 256 == 0, so an unaligned hidden size silently runs its matmuls on the CPU (Zamba-7B-v1 3712: 383/5113 nodes, BlackMamba-2.8B 1472: 145/1179) #991

Description

Measured, on three models that differ only in hidden size

Root cause

Why it was invisible until the strict probe

What would close it (orchestrator decision: both)

Reproducer

Not doing

Owner-facing consequence today

Activity

bong-water-water-bong commented on Nov 4, 2026

bong-water-water-bong commented on Nov 4, 2026

bong-water-water-bong commented on Nov 4, 2026

Part (b) attempt: the gate is not the anomaly — the decode schedule declares the same contract, so a gate-only relaxation is refuted (reverted, nothing shipped)

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions