hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9

Summary

On Qwen3.8-Flash-Next UD-IQ3_XXS (76.3 GiB), our HRX route loses three quarters of its decode rate on a
long context: 9.70 tok/s at an 18,782-token prompt against 31.15 tok/s for a llama.cpp fork built for
gfx1151, on the same GGUF, same requests, same hour. Prefill falls from 74–93 tok/s to 459. Short-prompt
decode is parity (28.6 vs 28.9), which is what makes this a long-context finding rather than a backend
finding.

Numbers

Protocol for every row: 512 greedy tokens (ignore_eos), median of 3, short 216-token / long 18,782-token
prompt, -c 32768 --parallel 1, one gpu-hold claim per arm, strixhalo gfx1151, 22:31–23:20 ADT.
Bold = the two rows being compared.

arm	config	short tg	short pp	long tg (18.8k)	long pp	GTT peak	peak RSS
c1 — our HRX route	house route's own server, `--device HRX0 -ngl 99 -np 1 --cache-reuse 256`, house defaults, no `-fa`, no MTP	28.61 (28.53/28.61/28.67)	465.7 (warm)	9.70 (9.23/9.85/9.70)	74 – 93	106.5 GiB	45.3 GiB
b1 — fork, no MTP	`-fa on -ctk q8_0 -ctv q4_0`, shipped default (its expert cache resolves off at this much device memory)	28.95	336.6	31.15	459.0	50.4 GiB	28.1 GiB
b3 — fork, `--moe-cache off`	same config as b1	31.72	350.9	31.14	465.5	50.4 GiB	28.1 GiB
c3 — fork + its cut-vocab MTP head, n=2	as b1 + drafting	44.86	329.8	47.87	458.4	52.2 GiB	28.4 GiB

Short prompt: 28.6 vs 28.9 — parity. With neither side drafting, the fork is not faster on this box.
Long prompt: 9.70 vs 31.15 — 3.2×, and prefill 74–93 vs 459 — 5–6×.
b1 vs b3 differ only in a flag whose value resolved off in both, so their 29–32 t/s spread is run-to-run
noise on this box, not an effect. The c1 numbers are medians of three, spread 9.23–9.85.

The configuration difference, stated so it is auditable

Our arm ran the house route's own server with the argv/env that 1bit serve --device hrx --print-backend
prints (engine hrxls): --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256,
GGML_HRX_REPORT=1, IREE_HAL_AMDGPU_LIBHSA_PATH=…, plus --no-strict-device because 1bit refuses
ffn_moe_gate-47 (MUL_MAT_ID, f32, ne=[640,10,0,1]) under strict placement — the house route leaves 2 of
6060 nodes on the CPU for this model. The fork's arms ran -fa on -ctk q8_0 -ctv q4_0.

The -fa on control on our route was never run. Arm d1 (-fa on -ctk q8_0 -ctv q4_0 on c1's exact
server) was killed during its load when the sequence was stopped, so there is no number for "our route with
flash attention at 18.8k". That is the one missing row, and it decides how the 3.2× is attributed:

if -fa on closes most of the gap → the finding is "the house defaults for this model must enable
flash attention at long context", and it is a one-line default, not a backend defect;
if it does not → the long-context attention path (not the backend as a whole, given short-prompt parity)
is where the work is.

What is already excluded by the data above: "the fork is faster" (it is not, at short context, without
drafting), and "the file or the quant is the difference" (same GGUF, byte-identical).

Why this matters

This is the first file where a published long-context number exists for the house route, and it is the
kind of row a user will hit first: a 76 GiB model at a 32k context. It also sits next to #572 (Flash-Next
prefill 285 tok/s on UD-IQ4_XS vs 1,679 for a tuned Vulkan engine), which is the same family at a short
context — different file, different mechanism, and the two should not be conflated: #572's blocker is a CPU
GET_ROWS node and f32 dispatch overhead; this one is decode attention over 18.8k of history.

Reproduce

# house route (one gpu-hold claim)
llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
    --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256 --no-strict-device
    # env: GGML_HRX_REPORT=1 IREE_HAL_AMDGPU_LIBHSA_PATH=<TheRock libhsa-runtime64.so.1>
# fork, same box, same file, same hour
llama-server ... -fa on -ctk q8_0 -ctv q4_0      # + the fork's own cached build for gfx1151
# client: 512 greedy tokens, ignore_eos, short 216 / long 18,782 tokens, medians of 3

Sampler: peak GTT from /sys/class/drm/card0/device/mem_info_gtt_used at 1 Hz, peak RSS from /proc.
Per-arm server logs, bench.json and the client sample.txt are under ~/lb/aval/*/ on strixhalo.

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572 — Flash-Next prefill on UD-IQ4_XS (short context, CPU GET_ROWS node); different mechanism.
The same arm produced two [gfxhub] page fault + MODE2 GPU reset bursts at GTT 106.5 of 120 GiB; filed as hrx: gfxhub page fault + MODE2 GPU reset when GTT nears the 120 GiB ceiling (Flash-Next UD-IQ3_XXS, 18.8k ctx, 106.5 GiB GTT) #680,
separately to keep this issue about throughput.

Attribution for #679 — long-context decode on HRX (Qwen3.8-Flash-Next UD-IQ3_XXS)

Everything below is measured on strixhalo (gfx1151), medians of 3 unless the row says otherwise, one gpu-hold
claim per arm, GTT sampled at 1 Hz with a hard 95 GiB abort guard.

1. The "one missing row" cannot be measured as written, and it is not the mechanism

Arm d1 (-fa on -ctk q8_0 -ctv q4_0 on c1's exact server argv) does not run: the server loads (49.7 GiB peak
GTT) and the first decode dies with

E graph_compute: unsupported HRX node: SET_ROWS output=621:q4_0[512,8192,1,1]<-SET_ROWS
E llama_decode: failed to decode, ret = -3

HRX's only quantized-KV writer (match_set_rows_addon, ggml/src/ggml-hrx/dispatch_registration/common/dispatch-rope-set-rows.cpp)
accepts GGML_TYPE_Q8_0 only, so a q4_0 V cache has no writer at all; and the HRX flash-attention matchers require
KV element size sizeof(ggml_fp16_t), while llama.cpp force-enables FA for a quantized V cache. So the fork's KV
configuration is doubly unreachable on our route, and -ctk q8_0 -ctv q4_0 would not use the HRX FA kernel even if the
write existed.

The -fa half of the difference is also a non-issue, measured on the same arm at -c 8192, 5.4k prompt, 64 tokens:

arm	flags	tg	pp
d2	`-fa on`	14.65	205
c1 argv	no `-fa` (llama.cpp default is `auto`)	15.91	443
c1 argv	`-fa off`	20.65 (short ctx only)	segfault during the 5.4k prefill

i.e. c1 already ran the HRX FA kernel, and turning it off is both slower and unstable.

2. What actually scales with context: the QSA selection path

Qwen3.8-Flash-Next's full-attention layers are indexer-sparse (QSA, src/models/qwen4exp.cpp). While
n_kv <= indexer_top_k + r - 1 = 2051 the top-k graph is skipped entirely (dense); above it, every such layer, every
step, runs: indexer gather over the whole indexer KV -> block pooling -> ggml_top_k(2051 of n_kv) -> a full-n_kv
mask rebuild -> flash_attn_ext.

One server, one process, decode 64 greedy tokens, median of 3 (-c 8192, f16 KV, no -fa override):

n_kv	pp t/s	tg t/s	ms/step
1,001	694.9	28.06	35.6
1,498	676.1	25.71	38.9
1,986	683.8	25.49	39.2
2,096	595.7	20.42	49.0
2,398	620.6	19.40	51.5
3,005	293.1	17.86	56.0
3,995	491.5	17.33	57.7
5,391	414.6	16.43	60.9
6,978	351.7	14.84	67.4

stepping from 1,986 to 2,096 tokens (the only change is crossing 2051) costs +9.7 ms per decode step;
above it the slope is ~3.8 us per KV token per step, which extrapolates to ~112 ms/step at 18,782 tokens against
the 103 ms (9.70 tok/s) that c1 measured there -- the whole gap to the fork's flat 31.15;
forcing the same path below the threshold (LLAMA_QSA_DENSE=0) at fixed n_kv ~1,480 costs 13x prefill
(676 -> 52.1 tok/s).

GGML_OP_TOP_K is the only op in that path with no HRX dispatch: it is absent from eager_capability_declared()
(ggml-hrx.cpp), no matcher registers for it, so device_supports_op is false, the scheduler puts it on the CPU
backend and splits every full-attention layer's graph around it. The CPU implementation is a single-row
std::partial_sort with an indirect comparator (ggml-cpu/ops.cpp), and QSA has one row per layer per step.

3. The fix and what it buys

PR 1bit-MONSTER/llama.cpp#161: GGML_OP_TOP_K on HRX as a radix select on a monotone uint32 key map (three levels
of 11/11/10 bits to the exact rank-k key, then one emit pass), one workgroup per row, 8 KiB LDS histogram, no global
scratch, no host round trip; plus a bounded claim in device_supports_op so shapes the kernel cannot take keep their
CPU fallback.

test-backend-ops -o TOP_K -b HRX0: 484/484 with ggml_hrx dispatch: hip.top_k_f32 (single), and the
GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32 control degrades to the CPU backend instead of failing the graph.

Long-context rows with the kernel, same binary, same pin, paired against GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32
(identical to the pre-PR behaviour):

arm	n_kv	tg t/s	pp t/s
kernel ON (`hip.top_k_f32`)	pending
kernel OFF (`GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32`, = pre-PR behaviour)	pending

Those rows are queued behind a release-gate window on the box (a 76.3 GiB Flash-Next load needs ~88+ GiB of
MemAvailable and the box is currently hosting other work), together with the 18,782-token protocol row (512 greedy
tokens, median of 3, -c 19456, 95 GiB GTT cap). I will post them here as a follow-up rather than guess.

What is measured for the kernel so far, at the op level (test-backend-ops):

-o TOP_K -b HRX0: 484/484, ggml_hrx dispatch: hip.top_k_f32 (single) in the dispatch log, and the
GGML_HRX_DISABLE_DISPATCH control falls back to the CPU backend instead of failing the graph;
perf -o TOP_K -b HRX0: 106 us (n=1000), 230 us (n=65,000), 623 us (n=200,000) per single row -- so the QSA decode
shape (12 layers, one row each, n=n_kv) should cost ~1.5-2 ms per step against the ~49 ms the CPU path costs at
18.8k. The fixed part is the serial 2048-bucket scan in the radix select, which the GEAK loop will attack next.

4. Not the cause (so no one re-does these)

flash-attention kernel coverage on the full-attention layers: matched and HRX-placed at f16 KV; no rejected nodes
in any server log of these runs.
KV layout / KV bytes: 462 MB of KV traffic per decode step at 18.8k is ~5 ms at 100 GB/s -- not the ~49 ms that
was missing.
The GDN / linear-attention state path: fixed-size state, 36 of 48 layers, no context term in the curve.
The 2 CPU MUL_MAT_ID expert nodes (--no-strict-device): they are per step, not per layer, and have no
context term.

5. Residual, after the kernel

The O(n_kv) mask rebuild that follows the selection (a [1,n_kv] f32 zeros + set_rows + add, then a cast for FA)
is still per-layer-per-step work that grows with context, on HRX but serial. If the numbers above leave a slope, that
is where it is, and the next step is either a fused qsa_mask_build dispatch or an FA variant that reads the selected
indices directly instead of rebuilding a dense mask.

Follow-up on #679 — the long-context rows with the fix in (and what it does not fix)

Same box (strixhalo, gfx1151), same model (Qwen3.8-Flash-Next UD-IQ3_XXS), one gpu-hold claim per arm, GTT sampled at
1 Hz under a 95 GiB abort guard, 18,782-token prompt, 512 greedy tokens, ignore_eos, median of 3.

The fix: fork PR 1bit-MONSTER/llama.cpp#161 — GGML_OP_TOP_K on HRX (radix select on a monotone uint32 key map of
the f32 bit pattern, one workgroup per row, 8 KiB LDS histogram, no global scratch, no host round trip) plus a bounded
claim in device_supports_op. test-backend-ops -o TOP_K -b HRX0: 484/484 with
ggml_hrx dispatch: hip.top_k_f32 (single), and GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32 degrades to the CPU backend
instead of failing the graph — which is what makes a clean pre/post comparison possible on one binary.

The rows

Both rows: same binary, same pin (4a58da9b0 = hrx-base + #159 + #160 + #161), same flags
(-c 19456 --no-repack, c1's argv otherwise). The only difference is the environment variable.

arm	short tg (216)	long tg (18,782)	long pp	peak GTT
`GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32` (pre-PR behaviour)	28.61	10.90 (11.02, 10.78)	131.0 (134.1, 127.9)	84.17 GiB
kernel in (`hip.top_k_f32`)	28.43	12.67 (12.56/12.94/12.67)	388.7 (388.7/411.3/251.4)	83.82 GiB
c1, published above (old pin, `-c 32768`, no `--no-repack`)	28.61	9.70	74–93 (84.5)	106.5 GiB

(the disabled row has two long reps rather than three: I cut it short to hand the GPU to two lanes already queued behind
my claim; the two agree to 2%, and the enabled row's three agree to 3%.)

What the dispatch buys, measured on one binary

	disabled	kernel	factor
long-context prefill	131.0 t/s	388.7 t/s	2.97x
long-context decode	10.90 t/s	12.67 t/s	1.16x
short-context decode	28.61	28.43	flat

Prefill is the win, and it is the expected one: the QSA layers call TOP_K once per layer per ubatch with one row per
token, so prefill was 512 serial std::partial_sorts on 4 CPU threads (with a graph split around every layer); on device
it is 512 workgroups. Long-context prefill now sits at 389 t/s, i.e. within reach of the gfx1151 fork's 459 rather than
5–6x under it.

Decode at 18.8k only moves 1.16x, so the selection was not where decode time went. What remains is the rest of the
QSA path, all O(n_kv) per full-attention layer per step: the indexer get_rows over the whole indexer cache, the
block→cell expand, and the [1, n_kv] f32 mask rebuild (zeros + set_rows + add, then a cast for FA) ahead of a
dense FA over all n_kv. Consistently, the 2051-token cliff survives with the kernel in (26.86 → 21.55 tok/s across it in
the same run): crossing 2051 switches the whole path on, not just the sort.

Correction on the memory side (#680)

I expected the ~24 GiB long-prompt GTT transient (c1: 106.5 GiB) to be the per-layer graph splits, and therefore to
disappear with the dispatch. It does not: the disabled row on the new pin/flags also peaks at 84.17 GiB. So the kernel
has no GTT effect (83.82 vs 84.17, noise), and the 106.5 → 83.8 GiB difference comes from the pin move and/or
-c 19456 + --no-repack, not from this PR. The operationally useful fact for #680 stands either way: at
-c 19456 --no-repack on the current pin, this model's 18.8k row never exceeds 84.2 GiB, ~11 GiB clear of the 95 GiB
rule, with or without the kernel.

Next, in order

The residual decode term — the O(n_kv) mask rebuild and indexer gathers, not the selection: either a fused
qsa_mask_build dispatch or an FA variant that consumes the selected indices directly instead of rebuilding a dense
mask.
GEAK on the new kernel — its fixed cost is the serial 2,048-bucket scan (~100 us/row); the plan is cooperative
scan / warp-aggregated histogram / 8-bit levels.

hyperloom custom: size the long-context prompt by what the tokeniser …

hyperloom custom: a long-context decode leg, so the tuner can score e…

TOP_K on HRX is merged on the fork (#161 → 199165047), but this issue stays open for two reasons, both stated so it is not closed on the merge:

it is in no build: engine main pins third_party/llama.cpp = 4a58da9b0, 8 commits behind the fork tip 199165047, and there is no open bump PR (#759 → 37ca4d442 was closed as superseded at 15:54:53Z).
the row still needs a fault-free arm: with the TheRock nightly libhsa, any run without HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 faults on its first graph replay (hrx: gfxhub permission page faults (UTCL2 client CPF) on the rebuilt box with the TheRock nightly libhsa — the no-SVM env trio is the only fault-free arm #715, gfxhub … UTCL2 client CPF), and test-backend-ops hangs on the fault. No fault-free measurement of the 18.8k row exists yet.

Closes on: the pin bump, then arm d1 (-fa on -ctk q8_0 -ctv q4_0) at 18.8k, memory-capped and alone, fault delta 0, beating the 28.43 tok/s short-prompt baseline.

Follow-up: the TOP_K kernel from #161 now has a 2.38x geomean improvement (op-level), branch 1bit/hrx-top-k-coop-scan ready.

The dispatch fixed the cliff and the prefill gap, but it left one measured defect: inside the radix select the descending
bucket scan ran serial in lane 0 — 3 levels x 2,048 buckets = 6,144 dependent LDS reads plus integer work,
independent of n. That is the ~100 us fixed cost test-backend-ops perf -o TOP_K -b HRX0 shows at n=1000, and it is
paid by all 12 full-attention layers on every decode step.

Each lane now owns 8 consecutive buckets; the 256 per-lane chunk sums get a reverse (suffix) inclusive scan as an LDS
Hillis-Steele tree (8 steps, 2 syncs each); the crossing lane is the one with suffix_incl >= remaining > suffix_excl,
and only that lane walks its own 8 buckets. Same bucket, same residue, same tie behaviour as the serial scan.

Same-hold A/B against the merged kernel, arms interleaved A/B/A/B, best of 3 rounds per shape, only that one .hip
differs:

shape	before	after	ratio
n=1,000, rows=1, k=400	111 us	13 us	8.74x
n=1,000, rows=16, k=400	108 us	14 us	7.48x
n=65,000, rows=1, k=400	246 us	155 us	1.50x
n=65,000, rows=16, k=400	341 us	258 us	1.29x
n=200,000, rows=1, k=400	643 us	511 us	1.25x
n=200,000, rows=16, k=400	908 us	766 us	1.16x

geomean 2.38x, worst 1.29x. The 111 -> 13 us at small n is the fixed cost disappearing; the n-proportional slope is
untouched, which is why the large-n rows only gain 1.16-1.25x. Correctness: -o TOP_K -b HRX0 484/484 with
hip.top_k_f32 matched, the GGML_HRX_DISABLE_DISPATCH control still declining every case as unsupported, CPU
reference 529/529. Registers 17 VGPRs (was 8), 0 spills, occupancy 16 waves/SIMD, LDS 9,228 B (was 8,204). Only the
kernel changed — the matcher, its shape bounds and the device_supports_op claim are byte-identical, and the add-on ABI
check still reports 336 kernels matching their code objects.

Two things worth recording for whoever touches this next:

Do not use __shfl_down_sync here. HIP rejects a 32-bit mask on this toolchain, and with a 64-bit mask an
unguarded add inflates every suffix sum at the top of the wave, because shfl_down returns the caller's own value
for an out-of-range source lane rather than zero: ERR = 4.0 on n=2/rows=2, ERR = 0.667 after adding the range
guard. The LDS tree is wave-agnostic and cheap; the shuffle is not worth the tuning.
The harness's "484/484" is 484 of 529 cases. The other 45 are shapes the bounded claim declines by design
(k > n, non-f32, n above the template range) and they correctly fall to the CPU backend — not failures.

Not measured, and it is the one thing left: the model row (18.8k decode, kernel ON vs
GGML_HRX_DISABLE_DISPATCH=hip.top_k_f32). It needs the 76 GiB Flash-Next file at -c 19456 with ~92 GiB of
MemAvailable and the owner's serve holds ~35 GiB of this 122 GiB box, so the 18.8k rows for #679 are still
memory-blocked. Every number above is op-level.

Branch 1bit/hrx-top-k-coop-scan @ 8b7966f90 (base 199165047, one commit, one file) is pushed and ready for a PR;
the round log with the measurement protocol (including why a single 81-shape sweep is not usable: a byte-identical
candidate "measured" 0.59x-2.05x on it) is at ~/geak-tasks/tattn-top-k/ROUNDS.md on strixhalo.

Addendum to the comment above — one more in-contract change is in the same branch: geomean 2.38x -> 2.77x.

The emit phase ran two full passes over the row (keys strictly above the rank-k key, then keys equal to it). The
split is already known when the level loop ends — exactly top_k - remaining keys are above, remaining are equal — so
both groups can be written in one pass into disjoint slot ranges, [0, above_slots) and [above_slots, top_k),
with the per-group slot guard keeping the result duplicate-free (the ties variant's requirement). Five passes over the
row become four.

shape	before (merged #161)	+ cooperative scan	+ single emit pass	vs before
n=1,000, rows=1	106 us	13 us	11 us	9.36x
n=1,000, rows=16	106 us	14 us	12 us	9.05x
n=65,000, rows=1	225 us	155 us	133 us	1.70x
n=65,000, rows=16	328 us	258 us	211 us	1.55x
n=200,000, rows=1	617 us	511 us	423 us	1.46x
n=200,000, rows=16	865 us	766 us	620 us	1.40x

geomean 2.77x, worst 1.40x, baseline's own per-round spread 1.01x (quiet box). Correctness re-verified on this exact
candidate: 484/484, control declining all 529, CPU 529/529, 17 VGPRs, 0 spills. Branch head is now 1b292a11b (still one
commit, one file).

One thing the measurement says that changes the next step, worth flagging rather than leaving for a later PR: the
remaining cost is parallelism, not atomics or bandwidth. The kernel reads the row 5 times (now 4), and at
rows=1 it moves ~8 GB/s on a ~100+ GB/s box while rows=16 reaches ~83 GB/s — the 16-row case does 16x the work in
1.54x the time, so throughput scales with workgroup count and rows=1 (one workgroup, ~1/40th of the GPU) is starved.
Splitting a long row across P workgroups is therefore the biggest remaining lever for the QSA shape — but it is not
doable inside the kernel: it needs a global histogram and a device-scope counter, i.e. either an extra binding for
scratch (the ABI is fixed) or a cooperative launch (the matcher launches one workgroup per row). That is a
matcher-level task, so it should be scheduled as one instead of being smuggled into the kernel work.

TOP_K on HRX, round 3 (vec4 row reads): 1.909x over round 1, 5.436x cumulative against the #161 pin. Branch
1bit/hrx-top-k-coop-scan @ b25b0485d (2 commits, 1 file, +101/-28, base 199165047). Op-level only — the 18.8k
model row is still unmeasured and is not being claimed.

Why vec4, and the number that pointed there

Round 1 (1b292a11b: cooperative bucket scan + single emit pass) removed the fixed ~100 µs and one pass, and measured
2.77x against the pin. I then tried candidate compaction (after level 0, write the higher-bucket indices straight to
dst and collect the selected bucket into an LDS candidate list, so levels 1/2 and the emit run over ≤2048 entries — 4
row passes to 2). It was rejected on measurement: 0.973x geomean, worst 0.95x against round 1, on a quiet box. The
passes it removes are cheap because the row is cache-warm by then; its restructured base was ~5-10% slower to start with.

That failure is the useful part: at rows=1 one pass is 200 000/256 = 781 iterations per lane and ~143 ns each
(~300 cycles) = one un-hidden DRAM latency per iteration, because a lane has one outstanding load and the workgroup is
only 4 waves. So the limit is memory-level parallelism, not work. Every pass (3 histograms + emit) now reads float4
when the row base is 16-byte aligned (src + row*column_count, so the guard covers pointer and pitch), with a scalar
tail for column_count % 4 and a fully scalar fallback. Index order is unchanged, so the ties rule and slot numbering
are untouched; the ABI, the matcher bounds and the device_supports_op claim are byte-identical to the pin.

shape	round 1	round 3	ratio
n=1 000, rows=1, k=400	12 µs	11 µs	1.03x
n=1 000, rows=16, k=400	12 µs	11 µs	1.07x
n=65 000, rows=1, k=400	127 µs	55 µs	2.30x
n=65 000, rows=16, k=400	211 µs	81 µs	2.61x
n=200 000, rows=1, k=400	414 µs	165 µs	2.51x
n=200 000, rows=16, k=400	622 µs	212 µs	2.93x

geomean 1.909x, worst 1.03x; cumulative against the pin on the same gate 5.436x, worst 4.23x. Protocol:
same-hold, arms interleaved A/B/A/B, best of 3 rounds per shape, raw per-round values printed (this box's per-sweep
noise is one-sided and reaches 1.6x, so a phase-ordered or single-sample protocol measures the box, not the kernel).

Gates (all re-run on the pushed head)

test-backend-ops -o TOP_K -b HRX0 484/484 with hip.top_k_f32 (single) taken; GGML_HRX_DISABLE_DISPATCH control
declines all 529 cases as not supported [HRX0] with nothing failing; CPU reference 529/529; whole-backend guard
identical 4289 per-case outcomes (the pin's own 320 [VIEW]/TOPMOE failures are present on both arms and their
error text is normalised). Compiler: 18 VGPRs, 34 SGPRs, 0 spills, occupancy 16, LDS 8 228 B. The algorithm and the
rejected design were both modelled in scripts/ref_topk.py against a sort reference first (616 cases, 0 failures).

What this does and does not mean for the #679 gap

There is no expected decode jump: top_k at 18.8k is ~12 x 50 µs ≈ 0.6 ms of a ~79 ms step (~0.8%), so the decode
residue stays where the attribution put it — the O(n_kv) mask rebuild and the dense FA over n_kv (P2 in the kernel
plan). Where it should show is prefill: the same dispatch was worth 2.97x there, and it is now ~5x faster at the
n=8192..262144 shapes prefill produces. The row that would confirm both (18 782 tokens, 512 greedy, median of 3,
-c 19456 --no-repack, kernel on vs GGML_HRX_DISABLE_DISPATCH) needs the 76.33 GiB qwen4exp file and ~92 GiB of
available memory; it is queued behind a headroom gate and no model-level number is claimed here.

Measured tonight: the attention path is ~14 % of this gap, not the gap

The 9.70 tok/s row is from c1 (pre-#161). Since then the QSA path has been fixed in three measured steps
(TOP_K on device #161, the mask rebuild's SET_ROWS, and now sparse index-list attention), so it is worth saying
plainly where the long-context decode time actually goes, because the answer is not the attention.

The measurement that settles it (new tonight, no model needed): the model's entire full-attention set for one
decode step — 12 QSA layers, each with indexer gather + pool + top_k(2051) + the mask rebuild + attention — built as
one graph at the GGUF's real shapes, and timed as one step:

n_kv	12 QSA layers, today's path (Loom decode split)	with sparse index-list attention (#49)
18,816	13.00 ms	9.95 ms (829 us/layer)
2,048	8.02 ms	9.74 ms → guarded back to 7.75 ms

So at 18.8k the whole attention path — the thing this issue was opened about — costs ~10 ms of a decode step. The
model decodes at 12.67 tok/s on today's main, i.e. ~79 ms/step, of which the short-prompt floor is ~35 ms; the
context-dependent slope is therefore ~44 ms, and the 12 full-attention layers account for 10 ms of it (~23 % of the
slope, ~13 % of the step). The other ~34 ms/step of context-dependent cost is not in these layers.

That matches the per-stage probes rather than contradicting them: indexer 0.60 ms, top_k 0.07, mask rebuild 0.07,
attention 0.81 per layer at 18.8k — and 12 layers in one graph cost exactly 12x that, so there is no hidden per-node or
per-split overhead inside the attention graph (the earlier "1.5 ms measured against 5.9 ms/layer implied" gap was me
dividing the model's whole slope by 12 QSA layers, which assumed the slope lives there; it does not).

What is left, with what each candidate needs to be settled

The remaining context-dependent cost has to be one of these, and every one of them needs the model to attribute:

The 36 GDN (linear-attention) layers. Their state is fixed-size, so this is expected to be context-flat — but it
is the largest unmeasured block in the step, and "expected" is not measured.
Per-step host/JIT overhead. This model needs 66 graph programs and the cache holds 64
(GGML_HRX_GRAPH_PROGRAM_CACHE, earlier sessions counted 933 trims in 40 min), so at least one program is evicted and
re-JITed per step. GGML_HRX_GRAPH_PROGRAM_CACHE=128 was measured to give 0 trims on another model and was never
measured here — the row never ran.
Memory pressure. The 18.8k arm peaked at 83.8 GiB GTT against ~50 GiB at 8k, on a box where GTT is DRAM-backed
and the weights are 76.33 GiB. A context-dependent fault rate is exactly the shape of a slope that no kernel-level
profile shows.

Why the model row still has not run

It needs the 76.33 GiB file with ~92 GiB MemAvailable (the arm's own GTT peak is 83.8 GiB). Tonight the box has
49-51 GiB available with the owner's forward-serve holding 31.7 GiB, and the waiter is armed for the moment that
clears. I am not lowering the gate to manufacture the row — the last time this arm ran near the ceiling it took the box
down (engine #680).

Ours, fixed tonight

gpu-kernels #49 @ 22dc9f6: sparse index-list attention for these layers (3.0 ms/step off the attention at 18.8k,
verified against the dense path at NMSE 1.7e-07), including the guard the 12-layer row exposed — at n_kv ~2048 the
selection is the whole cache and the same dispatch was 1.7 ms/step slower, so it now declines when the selection covers
half the cache or more.

What I would do next, in this order, once the box has the memory: (a) the same 18.8k row with
GGML_HRX_GRAPH_PROGRAM_CACHE=128 — one env var, and it either moves the step or kills candidate 2 for good; (b) the
same row with the 36 GDN layers' graph timed separately (they can be built model-free the way the QSA set was); (c) only
then a kernel, if what is left is arithmetic.

Long-context follow-up: multi-token prediction is not a long-context lever (paired A/B, one claim, both orders)

Different file from this issue's headline row, said up front so the numbers are not moved onto Flash-Next: this is
Qwen3.8-27B UD-Q4_K_XL (17.56 GB), 20,480 ctx, 18,762-token prompt, via the Hyperloom custom workload on strixhalo
(tools/hyperloom/custom/benchmark.py, engine 7d7c46f >= d3d2667a6, llama.cpp 1d32c226, hrx-system 82804197bcbe,
kernel 7.3.0-rc4-1bit-perfopt+). Metric: decode tok/s at depth (the serve long leg), plus the short serve legs.

Why paired, and why two orders

The arm-to-arm spread of this metric on this box is 19.3 % (12.938 / 15.435 / 13.342 on three identical no-lever
arms) -- larger than any lever effect measured here. So the lever was measured as A and B inside one gpu-hold claim,
then repeated with the order reversed, because "B second, onto a hotter die" is a real confound (the 2026-11-01 11:13
drafter fault reset the box).

pair (date)	order	A: no levers	B: `HL_MTP_MAX=6`	B vs A
1 (11-02 19:21-19:27)	AB (B second, die 43-64 C)	14.72 decode	13.29 decode	-9.7 % decode; prose -23.5 %, short -26.8 %
2 (11-03 15:19-15:27)	BA (B first, die 53-66 C)	14.5893 decode, geomean 48.8571	11.9825 decode, geomean 47.9897	-17.9 % decode; prose -23.6 %, short -27.3 %, code +3.1 %, geomean -1.8 %

Both halves of both pairs gate PASS at KLD 0 (greedy texts identical), gfxhub fault lines 1 -> 1 (MES banner only).

The sign did not flip, and the prose/short penalty replicates to within 0.7 pp across the two orders (-23.5/-26.8 vs
-23.6/-27.3). Running B first, onto a cooler box, makes the decode penalty larger, so the effect is the lever, not a
thermal ramp; the pair-1 ordering was masking part of it. The decode magnitude itself (-9.7 vs -17.9 %) is not stable
across pairs because the two halves' prefill differed by 27 % in pair 2 (250.85 -> 318.77, second half faster) -- and that
drift runs against the conclusion (the faster half is the one that won), so -17.9 % is conservative. Prefill is not
comparable across days at all (139.57 / 279.93 / 318.77 on the same binary), so decode is the metric of record here.

Mechanism, captured in the same runs: draft acceptance falls 0.6746 -> 0.4381 with depth (drafts 85/126 -> 92/210,
mean draft length 3.02 -> 3.63). Deeper drafts are converted less often, so at depth the lever spends draft work it cannot
turn into tokens.

What this means for this issue

HL_MTP_MAX=6 must not be carried onto long context, even though it is a +13.8 % win on the short-prompt geomean
(50.87 vs 44.69 clean). The two metrics disagree; only the short-prompt number stands.
The long-context gain this issue is after is not going to come from the config levers the tuning loop can sweep:
drafting is measured harmful (above); HL_DFLASH at long context is refused in committed code (hyperloom custom: refuse the drafter at long context (#764's GTT ceiling does not cover it) #773) because it is the
arm that faulted the graph on 2026-11-01; HL_UB/HL_B move the llama-bench legs but not the serve decode leg
(1bit serve rejects -b/-ub/-ctk/-ctv and sizes those per route); ONEBIT_HRX_DECODE_SPLIT=0 is documented as a loss
at ctx >= 512. That points at the kernels -- the attention work in this lane's own thread -- not at the tuner.
Number of record for the shape on the current binary: 14.59 tok/s decode at 18,762 tokens (no levers, pair-2 half A,
prefill 318.77). Use that, not the 14.72 from the older binary, and not the 12.94 or 15.44 single arms.

#679 follow-up comment (draft, 2026-11-03 night)

Posted by pi-attn (strixhalo). Numbers are from this box, this pin, one gpu-hold claim per row; nothing below is a
projection unless it says so.

Two things changed since the last comment, and one candidate is now dead. All rows are on the fork build
b2b545d06 (= base d8bd0ff60 + the top_k GEAK branch).

1. The "our route with `-ctk q8_0 -ctv q4_0`" row is runnable now — and it is 3x to 17x SLOWER

The 10-31 comment said that row could not run: a quantized V had no SET_ROWS writer at all (match_set_rows_addon
accepts GGML_Q8_0 only), so the graph was refused outright. hip.set_rows (PR-READY branch 1bit/hrx-set-rows)
supplies that writer, so quantized KV now runs end to end on HRX with 0 rejected nodes. Measured on Qwen3-4B-Q4_K_M,
-c 32768, 29,150-token prompt, 64 greedy tokens:

KV	fa dispatch	prefill t/s	decode t/s
f16	`hip.fa_prompt_f16`	979.5	25.26
`-ctk q8_0`	`hip.fa_prompt_q8`	57.1	14.77
`-ctk q8_0 -ctv q8_0`	`hip.fa_prompt_q8`	55.7	18.92

Same three at -c 8192 / 3,257 tokens: f16 1153.9 / 59.90, q8k 378.1 / 53.62, q8k8v 350.8 / 46.70.

So the answer to "is quantized KV the missing row" is no, by a wide margin: it costs 3.0x prefill at 3.3k and 17.2x
at 29k on this route, in the same binary, with the quantization itself doing the expected thing (KV bytes halved - the
f16 row also shows prefill falling from 1154 to 979 t/s over the same range, i.e. KV bytes are not the limit). This is a
kernel gap, not a policy one, and not explained by bytes: the q8 cache is half the traffic of f16, so the 17x is in
hip.fa_prompt_q8's own schedule (one workgroup per (query head, 32-query tile), each walking the whole cache and
dequantizing its chunk into LDS - so the cache is dequantized once per query tile per head), not in the quantization.
The KLD/needle A/B for quantized KV is moot at these numbers.

2. Long context is not generically slow on HRX — control row on a non-QSA model

Qwen3.6-35B-A3B-UD-IQ4_XS (17.7 GB, 3B active), same binary, 64 greedy tokens:

ctx	prompt_n	prefill t/s	decode t/s	ms/step
8,192	4,909	607.4	31.53	31.7
32,768	21,227	755.2	33.99	29.4
131,072	82,017	387.0	28.15	35.5

A 35B-A3B MoE decodes at 26-34 tok/s at 82k tokens (30-38 ms/step, 1.4 GiB of KV) where Flash-Next at 18,782 tokens
measures 79 ms/step. "HRX decode degrades with context" is not a general property of the route; the residual is
Flash-Next's own layer mix, which is why the model-free layer probes below are the right instrument.

3. Candidate killed: the HRX program-cache trims do not cost decode time

Flash-Next at 18.8k needed 66 programs against a 64-entry cache and showed 933 trims in a 40-minute run, so
GGML_HRX_GRAPH_PROGRAM_CACHE=128 was one of the three ranked candidates. Measured on Qwen3.6 at 82k with decode-only
reps (the first request pays the prefill, the next two reuse the KV), arms alternating so the box's 8-15%/hour drift
lands on both:

arm	trims	decode t/s (2 reps)
cache default (64)	99	31.47, 33.20
cache default (64), second server	99	30.97, 32.22
`GGML_HRX_GRAPH_PROGRAM_CACHE=128`	89	33.47, 33.35
`GGML_HRX_GRAPH_PROGRAM_CACHE=128`, second server	89	30.30, 31.58

Indistinguishable. Caveat stated so this is not over-read: that model does not overflow the cache the way Flash-Next was
counted doing, so this weakens the candidate rather than closing it on the real model. It also means the 128 setting is
not worth a model row before something else does.

4. The 36 GDN layers: 15.0 ms of the step, and an instrument that is not yet numerics-clean

~/wt/hrx-probe/gdn_layer.cpp rebuilds one GDN layer's decode step from src/models/qwen4exp.cpp
(build_layer_attn_linear + build_conv_state_at) at the GGUF's real shapes (state 128, K heads 16, V heads 48, inner
6144, conv 4, channels 10240, the file's own Q6_K projections) and sums N of them into one graph:

layers	HRX0 p50	per layer	CPU p50
1	582 us	582	913 us
12	5,597 us	466	8,043 us
36	14,973 us	416	21,866 us

36 layers = 15.0 ms of the decode step, context-flat by construction (fixed recurrent state).
Drift over a fixed 36-layer graph, 600 reps: first quarter 14,993 us vs last quarter 14,976 us, ratio 0.999.
With the 12 QSA layers at 10-13 ms and the ~13 ms short-context floor, ~40 ms of the 79 ms 18.8k step remains
unattributed, which is now the whole question.

The caveat, measured rather than assumed. That graph's output is wrong on HRX: 1,400 of 2,560 values are nonfinite
and the finite ones are 29 distinct garbage values repeated 40 times each (~2e8), byte-identically across processes,
and unchanged when I shrink the probe's leaf ranges to mild values. I stage-bisected it by copying each layer-0
intermediate into a leaf (the only reliable read-back on HRX):

stage	HRX nonfinite	max\|err\|/max\|ref\| vs CPU
`llm.ssm_conv` out	0	3.3e-3
v (convolved)	0	3.3e-3
recurrent-state gather (786k)	0	6.1e-3
`common.rmsnorm_gate` out (normed)	0	4.7e-3
gated (ngated)	0	5.8e-3
the activation into the closing `ssm_out` matvec	0	5.8e-3
layer output (`ggml_mul_mat(ssm_out, final)`, Q6_K x 1-token f32)	1400	-

So the recurrent math, the conv state, the state gather and the gated norm are all correct, and the corruption is in the
last node: a quantized-weight matvec with a one-token f32 activation. The shapes are not the issue either - I parsed both
GGUFs, and Bonsai-2-27B (qwen35) has the identical GDN config (group_count 16, time_step_rank 48, inner 6144, state
128) and runs correctly on HRX with the same fused kernel. Switching that closing projection from Q6_K to f32 (same
graph, same values, one env var) makes the output finite and equal to the CPU reference (rms 4.20265 vs 4.20273,
NMSE 1.8e-4) and removes kquant.mul_mat.decode_f32 from the dispatch list, while the same kernel on a leaf activation
is correct in the same graph. So the recurrent math is right and the single broken node is the quantized-weight matvec
whose activation comes from another fused dispatch rather than from a leaf - the shape of a missing activation staging,
not a coverage or shape problem.

What that costs the numbers above: the 15.0 ms is a dispatch/shape timing, and with the file's own Q6_K weights it is
timed on a graph whose last node mis-stages its activation. I am quoting it as provisional, not as a validated row.
The ~40 ms unattributed arithmetic is unchanged either way.

And one thing this lane should say out loud about its own evidence: every number in this issue and in my earlier
comments is a speed row - the 76 GiB model has been memory-blocked for four days, so no KLD, needle or even
coherence check has ever been run on Flash-Next on HRX. If the model's GDN layers take the path found broken above (in
the model the projection may instead be fused into the GDN dispatch - ..._projection_epilogue exists in the kernel
catalog, and GGML_HRX_LOG_DISPATCH=1 on a real load would answer it in one line), the model would still generate tokens
at 12.67 tok/s and nobody would notice. Before the next parity number is quoted, the speed row and a quality check should
be taken in the same window.

Not run, and why: the 18.8k model row. Its own waiter logged 18:27 -> 19:50 on pool >= 88 GiB and
MemAvailable >= 92 GiB; the pool never exceeded 72 GiB and MemAvailable peaked at 66 GiB, because the owner's
1bit forward-serve holds ~30 GiB of GTT. The gate is not being lowered: the one attempt near the ceiling was the
#680 reset.

The long-context config surface is empty, and here is the log evidence — so nobody spends a GPU window on these two names.

This lane measured the 27B UD-Q4_K_XL at 20,480 ctx (HL_LONGCTX=18800, 18,762 tokens as the server counted them). Its numbers of record: 14.5893 tok/s decode (pair-2 half A, one claim) — not Flash-Next, not the 12.67 of the issue. Two sweepable names were left untested at long context (GGML_HRX_GRAPH_PROGRAM_CACHE, GGML_HRX_Q8_PREFILL_RELAX); both are now ruled out from the artifacts rather than by another arm:

The graph-program cache never trims in the long arm. grep -c 'graph program cache trimmed' over all 11 leg logs (serve.log, llama-bench.log, kld.log) of the four pair-2 arm dirs → 0 in every one, including the arm that did the 18,762-token prefill and the 128-token decode. A lever that sets the count cap to 0 (graph_cache_unbounded) cannot move a cache that never evicts, and it does not touch the separate 8 GiB byte budget (plan_evictions(budget, max_programs, …) applies both; GGML_HRX_GRAPH_PROGRAM_CACHE_MB defaults to 8192 and is not in the grid).
Trims happen in the KLD/perplexity leg, not the scored legs: 163 / 164 / 92 in the HL_UB=1024/2048 explore runs — trimmed (1 of 39 programs, 8583 -> 8323 MiB of arenas, budget 8192 MiB, 64 max). A quality-leg cost at widened chunk shapes; no score depends on it.
q8_prefill_relax_off is a prefill-routing knob (docs/hrx.md: prompt matmuls on the q8_1 x4 kernel, 94 % of HRX prompt time, prompt chunks 256–2048), and =0 restores the older, slower routing. The long-context score here is the decode rate, so it can only cost the 320 tok/s prefill and cannot raise decode.

Combined with what this issue's thread already has — HL_MTP_MAX=6 measured harmful at depth in both orders (−10…−18 % decode, −24…−27 % prose/short, draft acceptance 0.6746 → 0.4381), HL_DFLASH refused by committed code (#773) after faulting the HRX graph at 18.8k on 2026-11-01, and HL_UB/HL_B/-ctk/-ctv not reaching 1bit serve at all — there is no configuration lever left for the 18.8k decode rate. The remaining gain is attention-kernel work, which is not this lane's.

Separately, so the artifacts above are attributable from now on: #947 → PR #948 adds bench_summary.device_stack = {kernel, hsa_runtime, hsa_runtime_sha256} to every measurement, because the report named the binary (#933) but not the kernel, and this box rebooted five times in 18 h.

hyperloom custom: refuse a long-context leg that does not fit the pin…

#679 follow-up (pi-attn, strixhalo): the re-JIT price is measured, and it is the first thing in this issue with a number in the right order of magnitude

Model-free work only; the 18.8k row is still blocked (see the end). One number matters here:

A graph-program re-JIT costs ~41 ms, and it is flat in graph size

Measured by timing the first graph_compute of a fresh graph against the immediately following one - the second is
served from the HRX program cache - same graph, same leaves, one gpu-hold claim, 0 faults:

graph	nodes	first	second	price of a re-JIT	steady p50
1 GDN layer	44	41.9 ms	0.94 ms	40.9 ms	0.65 ms
12 GDN layers	528	52.0	6.04	46.0 ms	5.39 ms
36 GDN layers	1584	58.4	15.68	42.7 ms	15.29 ms
36 layers, cache limit 1, twice	1584	56.8 / 59.6	15.8 / 18.5	41.0 / 41.1 ms	15.5 / 15.3 ms

It is compile+load overhead, not per-command emission, so any single trim costs the same whether it is a small decode
program or a large prefill one; and it is 3-4x the whole steady decode of a 36-layer GDN graph. (First compute in a
cold process is 110-136 ms; 41 ms is the warm figure, cross-checked by the steady p50 in those same runs scaling with
the clock ramp and the JIT delta scaling with it.)

Why this matters for this issue, and what it does not say

This model's own 18.8k server log carries 933 trim_graph_programs ... (1 of 66 programs, ... 64 max) lines in one
40-minute run, and 779 in the pre-rebuild run of the same row; each trim evicts one program and frees ~6 MiB, so the
8192 MiB byte budget is never the constraint. Candidate (a) was "raise GGML_HRX_GRAPH_PROGRAM_CACHE; one env var, and
it either moves the step or kills the candidate" - ranked on intuition because nobody had priced a re-JIT.

It is priced now, and it sits inside the unexplained ~44 ms/step: at the observed rate of ~1 trim per 10 decode steps it
is ~4 ms/step; at 1 trim per step it would be ~41 ms/step, i.e. the whole residual. Which of those it is, is
still unmeasured, because it needs the 18.8k row itself.

So I shipped the cheap half as data rather than guess: engine #1006 adds one built-in recipe,
hrx-graph-program-cache-128 (hrx + qwen4exp -> GGML_HRX_GRAPH_PROGRAM_CACHE=128), with the trim counts and this
price table in its measured field and an explicit "the tok/s row is not measured" sentence. It is a recipe and not a
default change for the same reason. Raising the count does not preallocate: the byte budget still caps the cache, and
128 slots of ~6 MiB are ~768 MiB against a 120 GiB GTT budget.

One retraction that removes a red herring from this issue's history

The "GDN layers produce 1400/2560 nonfinite, diverging from CPU" rows were my probe, not the engine: gdn_layer.cpp
bound every tensor in one usage-ANY buffer, so the packed Q6_K weight layout was requested and never repacked - the
same class as the retracted #969. With the weight tensors in their own WEIGHTS-flagged buffer the fused GDN graph is
non-finite-free (0/2560) and its timing rows stand unchanged. A bounded difference from CPU remains (rel_rms
~9 %, no NaN) which I am not claiming as a defect: the delta-net recurrence runs 128 steps in f16 on device against f32
on CPU. The next step if anyone picks it up is ~/lb/tattn-gdn-f32.sh (f32 vs Q6_K projections through the same
comparison), which separates the quantised path from the fused op.

Still the same block

The 18.8k row needs the 76.33 GiB file with the box to itself: the load alone measured >= 109 GiB of GTT on
2026-11-04 (against a 120 GiB ceiling and #680's 106.5 GiB reset regime), and this box has carried other tenants' serves
(32-40 GiB) continuously since 10-31. Nothing in this comment is a substitute for that row, and none of it is a tok/s
claim.

recipe: flash-next gets 128 graph programs, because a re-JIT costs 41…

GDN path: settled by the owner's call — it is at CPU parity end-to-end

The orchestrator answered my last comment with "fetch a small GDN model (<= 5 GB) and settle the GDN quality question
first", so I did, and the answer is clean. I had flagged a 16 % relative rms (f32 projections) / 39 % (Q6_K) between the
fused GDN graph and the CPU reference on a fixed synthetic input, and said it was either f16 amplification over the
128-step recurrence or real precision loss, unsettleable from an aggregate rms.

Model: Qwen3.5-4B-Q4_K_M — qwen35, 32 blocks, full_attention_interval = 4, i.e. 24 of 32 layers are the GDN
path (same architecture family our recipes target; not a proxy). 2.74 GB, sha256 verified against the repo's LFS oid.
Same GGUF, same corpus, same flags, one gpu-hold claim:

row	HRX0	CPU	delta
PPL chunk 1	6.0473	6.0476	-0.0003 (0.005 %)
PPL chunk 2	8.5787	8.5790	-0.0003 (0.003 %)
greedy 24 tokens x 3 prompts	identical	identical	0 disagreements

And the path was really taken, which is what makes the row mean anything: the HRX log carries
llm.gated_delta_net.f32_wmma_head128 (fused) for the prefill/PPL graphs and hip.q36.gdn_decode (fused) for decode —
different matchers, both present.

Conclusion: the probe difference was a synthetic-input artifact — a ramped random input puts the recurrence in a
regime where f16-vs-f32 differences compound, and trained weights do not. The GDN/linear-attention half of Flash-Next is
therefore not a quality risk on this route. Reproducible with ~/lb/tattn-gdn-ppl.sh.

What this does not say: Flash-Next itself still has no quality row — that file needs the box to itself (load alone
measured at >= 109 GiB of GTT against a 120 GiB ceiling), so the claim is "a 24-layer GDN model is at parity", not
"Flash-Next is at parity". The Flash-Next-specific risks that remain are the f16 KV layout at 18.8k, the QSA sparse path
at long context, and the 76 GiB residency profile — all of them still gated on that same window.

recipe: flash-next gets 128 graph programs, because a re-JIT costs 41…

hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679

Description

Summary

Numbers

The configuration difference, stated so it is auditable

Why this matters

Reproduce

Next

Related

Activity

bong-water-water-bong commented on Oct 31, 2026

Attribution for #679 — long-context decode on HRX (Qwen3.8-Flash-Next UD-IQ3_XXS)

1. The "one missing row" cannot be measured as written, and it is not the mechanism

2. What actually scales with context: the QSA selection path

3. The fix and what it buys

4. Not the cause (so no one re-does these)

5. Residual, after the kernel

bong-water-water-bong commented on Oct 31, 2026

Follow-up on #679 — the long-context rows with the fix in (and what it does not fix)

The rows

What the dispatch buys, measured on one binary

Correction on the memory side (#680)

Next, in order

bong-water-water-bong commented on Nov 1, 2026

bong-water-water-bong commented on Nov 2, 2026

bong-water-water-bong commented on Nov 2, 2026

bong-water-water-bong commented on Nov 2, 2026

Why vec4, and the number that pointed there

Gates (all re-run on the pushed head)

What this does and does not mean for the #679 gap

bong-water-water-bong commented on Nov 3, 2026

Measured tonight: the attention path is ~14 % of this gap, not the gap

What is left, with what each candidate needs to be settled

Why the model row still has not run

Ours, fixed tonight

bong-water-water-bong commented on Nov 4, 2026

Long-context follow-up: multi-token prediction is not a long-context lever (paired A/B, one claim, both orders)

Why paired, and why two orders

What this means for this issue

bong-water-water-bong commented on Nov 4, 2026

#679 follow-up comment (draft, 2026-11-03 night)

1. The "our route with -ctk q8_0 -ctv q4_0" row is runnable now — and it is 3x to 17x SLOWER

2. Long context is not generically slow on HRX — control row on a non-QSA model

3. Candidate killed: the HRX program-cache trims do not cost decode time

4. The 36 GDN layers: 15.0 ms of the step, and an instrument that is not yet numerics-clean

bong-water-water-bong commented on Nov 4, 2026

bong-water-water-bong commented on Nov 4, 2026

#679 follow-up (pi-attn, strixhalo): the re-JIT price is measured, and it is the first thing in this issue with a number in the right order of magnitude

A graph-program re-JIT costs ~41 ms, and it is flat in graph size

Why this matters for this issue, and what it does not say

One retraction that removes a red herring from this issue's history

Still the same block

bong-water-water-bong commented on Nov 4, 2026

GDN path: settled by the owner's call — it is at CPU parity end-to-end

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. The "our route with `-ctk q8_0 -ctv q4_0`" row is runnable now — and it is 3x to 17x SLOWER