[bong-water-water-bong]

What

Closes the residual decode gap above key_value_token_capacity 2048 that #13 left behind (1bit-MONSTER/engine#124).

@ggml.flash_attention.decode_split.reduce_completed.multipass ended with a scalar output pass: workitem -> one output channel, then a serial scf.for %block = [0 to active_block_count] over every KV block, executed by all 256 workitems. Only 128 are live at value_head_size = 128, and each one recomputed scale = expf(partial_max[block] - maximum) for every block — expf once per (block, element) instead of once per block, with every element walking the block dimension alone. That pass was the entire ~16% gap.

Changes

• LDS scale stage (flash_attention_decode_split_f32_f16_wmma.loom): the lane-strided sum pass already computes the per-block scale; store it into a per-row workgroup stage (scale_stage_view[row, block]) and publish each row's sum (sum_stage_view[row]). The output pass reloads the exact f32 value, removing ~128x redundant expf and the redundant partial_max reloads.
• Vectorised all-rows output pass: after the per-row max/sum phase, a single pass covers all query rows — workitem w owns a 4-channel quad (row = quad / quads_per_row, channel = (quad % quads_per_row) * 4), loads vector<4xf16>, and accumulates vector<4xf32> over blocks with unroll(%c4) schedule(interleaved). At the production GQA shape (8 query heads per KV head, value_head_size = 128) there are exactly 256 quads, so all four subgroups stay busy.

The per-channel accumulation order over blocks is unchanged and the vector ops are elementwise, so the reduce output is bit-identical to the previous reducer.

Verification (gfx1151 Strix Halo, Qwen3-Coder-30B-A3B-Instruct-Q4_K_M, -dev HRX0)

llama-bench -p 0 -n 8 -r 5 -d 1900,2000,2100,3000,4800, median of 6 interleaved A/B rounds:

depth	capacity	blocks	base t/s	patched t/s	delta	path
1900	1920	30	70.72	71.48	+1.1%	cooperative (control)
2000	2048	32	66.06	68.82	+4.2%	cooperative (control)
2100	2112	33	58.19	67.15	+15.4%	multipass
3000	3008	47	50.84	60.89	+19.8%	multipass
4800	4864	76	41.83	51.02	+22.0%	multipass

The cliff is gone: d2100/d2000 0.881 -> 0.976 (d4800/d2000 0.633 -> 0.741). The two untouched depths moved +1.1% / +4.2% (run-to-run noise).

• Bit-exact: llama-perplexity -c 2049 -b 1 --save-all-logits (token-by-token decode; GGML_HRX_LOG_DISPATCH=1 confirms flash_attention_decode_split); the 1,244,725,284-byte logits dumps are byte-identical (cmp), PPL 21.9760 +/- 0.81813 on both.
• Correctness: buried code word ZX-4718-QQ at 4700 prompt tokens (capacity 4864, multipass), llama-server, temperature 0, seed 42, cache_prompt:false -> exact PASS.
• Faults: 6 A/B rounds x 5-rep sweeps at all five depths: 0 HSA_STATUS_ERROR_MEMORY_FAULT, 0 res = -3.

Full method, table and repro commands: benchmarks/NOTE-hrx-124-multipass-output-2026-09-27.md.

A throwaway diagnostic that deleted the output block loop (keeping max/sum) measured +21.6% / +50.0% / +30.5% at d2100 / d3000 / d4800 — the basis for attributing the gap to the output pass rather than the reduction math.

[bong-water-water-bong]

Review: the change looks right for the measured range: expf once per (row, block), a vectorised output pass, the #123 pack barriers untouched, and it builds on 00adc2b. One blocker before merge: LDS scales with context length.

scale_stage is query_heads_per_key_value_head × active_block_count × 4 bytes of workgroup memory. The multipass variant admits active_block_count up to 4096 (range(%active_block_count0, 33, 4096), i.e. 262,144 KV tokens), and query_heads_per_key_value_head up to 16:

rows (GQA)	blocks	KV tokens	scale_stage
8 (Coder-30B)	1024	65,536	32 KB
8	2048	131,072	64 KB
16	1024	65,536	64 KB
16	4096	262,144	256 KB

gfx1151 has 64 KB of LDS per workgroup, and this is the fused reduce_fused kernel, so the produce stages' LDS (query/score/probability/product stages) comes on top. The PR was measured up to d4800 (76 blocks, ~2.4 KB). Qwen3-Coder-30B's native context is 256K, and ZAYA is served at 64K.

Please either:

1. bound it: keep the scale stage for active_block_count ≤ N (N such that rows·N·4 plus the produce stages fit) and fall back to the old per-element path above that, or tile the output pass over block chunks with a fixed-size stage; or
2. show what happens past the limit: llama-bench -dev HRX0 -p 0 -n 8 -d 32768,65536,131072 on Coder-30B (guard the load with scripts/mem-guard.sh), plus a GQA-16 model if one is at hand. The acceptable outcomes are the matcher declining or a clean JIT refusal with fallback, not a fault or wrong output.

Smaller: quads_per_row = value_head_size / 4 silently drops channels when value_head_size % 4 != 0. Please add that to the matcher or assume it in the kernel.

[bong-water-water-bong]

Long-context check after merge (Qwen3-Coder-30B-A3B Q4_K_M, GQA 8 rows, HRX0, llama-bench -p 0 -n 8 -r 1, one run each under scripts/mem-guard.sh, 0 new GPU page faults):

KV tokens	this PR (tok/s)	before (tok/s)
32768	43.3	10.8
65536	43.1	36.8
131072	not measured (mem-guard stopped both at the 12 GB floor, KV + other load)	same

So up to 65K tokens at 8 rows (scale stage ~32 KB) it runs and is faster. The speed only; correctness at long context was not checked here. The LDS bound concern remains for >=128K at 8 rows and >=64K at 16 rows; a guard in the matcher (or a chunked stage) is still worth adding. Also: agent-872bf5's ZAYA1-8B bench on this build (bin180-new, -d up to 16384) has been stuck for over 1.5 h with the GPU idle and the HRX queue thread spinning. It is unexplained so far and might not be this change; see the owner's thread.

[bong-water-water-bong]

Follow-up on the ZAYA hang I mentioned: it does not reproduce. The identical llama-bench (bin180-new, zaya1-8b-Q4_K_M, HRX0, -p 0 -n 16 -r 2) ran each depth separately under a 600 s timeout on a quiet box after the 2026-09-28 reboot, and every run finished. Decode tok/s, split on / off: d2100 35.2/35.7, d3000 38.5/32.8, d4096 37.0/30.2, d8192 8.9/23.1 (the on-run's dip is likely first-use JIT at that shape), d16384 23.9/12.0. The earlier stall (GPU idle, HRX queue thread spinning) stays unexplained, but it isn't a repeatable property of this change.