HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines)

Split out of #390 (whose reporting half is done). With the HIP add-on the Flash-Next experts run on HRX (fork #142/#144, gpu-kernels #38/#39), and the file now measures PPL 1.8090, prefill 287 tok/s, decode 24.0 (UD-IQ4_XS, HRX0, no drafter; UD-IQ3_XXS 297 / 26.9). The field on the same chip reports prefill 510 (tuned Vulkan llama.cpp at 8k), 1,175-1,309 (halogen), 1,679-2,013 (gufo), 2,372 (hipfire), and decode 33-60 with drafters.

What gufo does that we do not (from the 2026-10-11 comparison, memory gufo-benchmark): prefill batches up to 4096 (dense) / 8192-token chunks (MoE), WMMA quant GEMMs or dequant-to-BF16 hipBLASLt for big shapes, FP16 activations, WMMA attention, MMQ experts with fitted tile widths, int8 WMMA for Q8_0 dense. Our expert prefill path is the HIP mmid GEMM at m64 tiles; the IQ2_S kernel GEAK authored is latency-bound at ~135 GB/s (52%).

Ask: a Flash-Next prefill plan on HRX: (1) dispatch profile of pp512 / pp4096 on UD-IQ4_XS with owners and the CPU splits (engine #389: -ub > 512 fails res=-3 on HRX, so chunking is capped), (2) expert-GEMM tile width fitted to the 512x640x2560 routed shapes, (3) the ub cap, (4) an MTP head on HRX for Flash-Next for the decode side. Numbers to beat: 1,679 prefill (gufo, same file), 33 decode (Vulkan llama.cpp with MTP).

🤖 Generated with Claude Code

docs: where Flash-Next prefill's 287 tok/s goes, and the ubatch cap t…

The claim still holds at the current pin, and the numbers are now in-tree rather than in a memory file — docs/npu-leaderboard.md:27-29:

Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill pp512  | 286.63 tok/s | ngl=99, fa=1, ub=512,  engine=0d0c846
Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill p1024 ub576 | 269.34 tok/s | ngl=99, fa=1, ub=576, engine=0d0c846
Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill pp4096 | 260.59 tok/s | ngl=99, fa=1, ub=512,  engine=0d0c846

So 287 tok/s vs gufo's 1,679 is real and reproducible on paper, and nothing in the tree since has moved it: the #622 tiled transpose and the #623 first-token pair are 27B NPU rows (435.9 s -> 145.7 s, docs/npu-leaderboard.md:34-35), not this prefill path. The gap analysis in the issue body (batch size, WMMA/dequant GEMM, FP16 activations, MMQ experts, int8 WMMA for Q8_0, latency-bound IQ2_S expert kernel at ~135 GB/s) is the right shape and is unimplemented.

Classification: needs GPU measurement — specifically llama-bench -fa 1 -ngl 99 -ub 512 -p 512 (3 repeats, and -ub 576 -p 1024 plus -p 4096 to match the three rows) on UD-IQ4_XS with the HIP add-on, on the current pin and again on 40c4ccb76e48 (the #610 bump target), reported as a table against 286.63 / 269.34 / 260.59. Without that row the next change to the expert GEMM has no before-number to beat. Left to the measurement lane — this box is CPU-only today.

Two things, and the second one is a retraction.

(a) Where this should look next. ggml_mul_mat_f32_f32_wmma is 8.6% of device time at pp512 — 56.8 ms of the pass — over 522 dispatches in the profiled bundle (docs/hrx.md:414-422), inside a 4.25 bpw model. That is a defect regardless of the per-layer accounting. One correction to the mechanism, though: it is not "some projection is falling back to an unquantized matmul" (docs/hrx.md:429-430). The operands are the file's own F32/BF16 tensors — 48 ffn_gate_inp (F32 [2560,512] router), 96 hc_{attn,ffn}_inject (F32 [10240,4]; N=4, so every quantized matcher declines), 72 ssm_alpha/ssm_beta (F32 [2560,48]), 24 indexer.{q,k}_proj (BF16) — 240 of the 261 per-forward f32 matmuls (perf-572-findings.md). ~95 GFLOP inside 56.8 ms ≈ 1.7 TFLOPS against ~9.5 for the expert GEMM, so the cost is dispatch overhead on tiny shapes, not a bad matmul. The lever is fusing or dispatch-batching those, not quantization.

(b) Retraction. An earlier reading today held that the bundle contains two forwards (92 swiglu dispatches = 2 × 46 layers) and therefore that device time was 37% rather than 76% of the pass. That does not survive checking. The GGUF says qwen4exp.block_count = 48 (header read of Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf, also qwen4exp.expert_count = 512), and the dispatch counts divide by neither 48 nor 46 — 92, 86 and 522 are 4×23, 2×43 and 2×261. So docs/hrx.md:409-411 stands as written and the 76% (1,355 ms of the 1,786 ms pp512 pass) is not being changed. The per-layer attribution inside the bundle is unresolved, not settled in the other direction either.

Next: the profiler run that settles per-layer counts — dispatch counts resolved against block_count with the bundle's own op names, at ub 512 and again at ub 2048, so kernel counts can be attributed per layer instead of divided by assumption.

Item 1: the host gap, measured. It is not there on any model that fits on this box, and the ~1.0 s/forward premise does not survive either instrument I could bring to it.

First the premise. The brief takes device time as 755.7 ms of the 1,786 ms pp512 pass (41.9 %) and the rest as host. This file's own profile records 1,355 ms of 1,786 ms as device (76 %) (the section merged as #616), and comment (b) above re-affirmed that 76 % after retracting the two-forwards reading. Those two splits cannot both be quoted: at 76 % the host leg is 431 ms, not 1,030 ms. The 755.7 ms figure is not in this issue or in docs/hrx.md.

Two instruments, both on engine 0d0c846 + llama.cpp 8ee20a0 + gpu-kernels 2b626798a.

A host-path profiler (GGML_HRX_HOST_GAP=1, llama.cpp ggml-hrx: an env-gated host-gap profiler (engine #572) llama.cpp#154, our own new file, off by default) that times every phase of a graph compute. Fully offloaded, llama-bench -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0, steady-state forward:

model	forward	graph build (outside the backend)	lookup	bind	upload	flush	wait (device)
Qwen3-Coder-30B-A3B Q4_K_M, 1,161 dispatches	263.5 ms	1.5 ms	0.004 ms	0.002 ms	0.005 ms	0.001 ms	263.46 ms
Qwen3.8-27B UD-Q4_K_XL, 1,519 dispatches	1369.0 ms	0.9 ms	0.003 ms	0.003 ms	0.004 ms	0.001 ms	1368.89 ms

Every item the brief names is in that row: graph build 1.5 ms, program-cache lookups 4 µs, host copies 5 µs, GET_ROWS fallback absent (one graph, no split, no CPU node), wait spin inside replay.wait where the device leg is. Total host: 1.5 ms of 263.5 ms, 0.6 %.

libhrx's own dispatch profile (HRX_PROFILE_FILE=<bundle> HRX_PROFILE_MODE=dispatch, then iree-profile statistics) — the only profiler that sees HRX dispatches:

model	dispatches in bundle	device kernel time	wall for the same 2 forwards	device/wall
Qwen3-Coder-30B-A3B Q4_K_M	2,322 = 2 × 1,161	538.14 ms	523.8 ms	103 %
Qwen3.8-27B UD-Q4_K_XL	3,038 = 2 × 1,519	1,871.15 ms	1,657.2 ms	113 %

Both instruments agree on the dispatch counts to the unit (1,161 and 1,519), and device kernel time is the whole forward; the >100 % is the profiler writing records.

rocprofv3 cannot do this job, and that may be where the 41.9 % came from. rocprofv3 --kernel-trace --stats on an HRX run exits 0 and writes no trace file: HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets, so the profiler's HSA interposition never sees a dispatch. Anything summed from a "device trace" that returns nothing, or from a partial one, gives an inflated host leg.

What a host leg looks like when there is one. The same 27B at -ngl 20 (a third of the layers on the CPU) runs 3,746 graph computes per forward instead of 2, and its host-staging upload alone costs 145.7 ms per forward (39 µs per graph), with the CPU backend's own compute on top. That is the shape of a real host component: a graph that is not fully on HRX.

Counts the last comment asked for (GGML_HRX_HOST_GAP=1, dispatches per forward):

model	p/ub 512	p/ub 1024	p/ub 2048
Qwen3.8-27B UD-Q4_K_XL	1,519	1,616	1,813
Qwen3-Coder-30B-A3B Q4_K_M	1,161	—	1,137

Resolved against block_count: 27B 62 layers → 24.5 dispatches/layer at ub 512, 29.2 at ub 2048; 30B-A3B 48 layers → 24.2 and 23.7. The counts are stable, so per-layer accounting can be done from them, but the timings are not: the same 27B pp512 read 828.6 / 870.2 / 1120.8 / 1369.9 / 1518.2 ms across five runs on this box today (±57 %). Any before/after on this lane needs repeats inside one gpu-hold claim.

What I could not measure. Flash-Next UD-IQ4_XS is 93.7 GB (87.3 GiB) of weights; this box has 75–82 GiB available with the owner's 27B NPU job resident, so loading it would swap and every number from it would be a swap number. UD-IQ3_XXS (76.4 GiB) is the one Flash-Next file that could be tried here, and only with that job stopped — your call, not mine. One run of the profiler on either file answers this issue's item 1 for Flash-Next itself.

On your item (a) (the f32 matmuls), from the same profile: on the 27B they are fused — common.mul_mat.f32_f32_wmma appears in the matcher census as fused and no ggml_mul_mat_f32_f32_wmma kernel row appears at all — so the 522-dispatch / 8.6 % shape is Flash-Next's own (hc_*_inject N=4 each getting a dispatch). The batching fix already has a precedent in this tree on a file that fits: qwen3_moe_router_projection_f32_four_row_wave32 batches the 30B-A3B's f32 router projection — 16 dispatches per forward, 3.3 % of its device time, instead of one per token.

PRs, neither merged: #646 (docs/hrx.md, the whole profile above) and 1bit-MONSTER/llama.cpp#154 (the profiler).

Follow-up with the long-prompt case closed, since it was the one place the host path did grow.

-p 4096 -ub 2048 on Qwen3-Coder-30B-A3B Q4_K_M, steady-state graph, every step named (llama.cpp #154 gained the remaining scopes: cache.key, replay.total, replay.rebind, replay.transient, replay.launch, replay.download):

wall 1158.581 ms = graph.lookup 0.179 + graph.bind 0.016 + graph.execute 1158.385
  execute          = cache.key 0.007 + replay.total 1145.534 + ~12.8 ms (cache mutex/prologue)
  replay.total     = replay.wait 1143.718 + rebind 0.002 + transient 0.001 + upload 0.004
                     + launch 0.042 + flush 0.001 + download 0.002

so host time outside the device wait is ~15 ms of a 1158.6 ms forward (1.3 %), and at ubatch 512 it is 0.03 ms of 263.5 ms (0.01 %). The 145 ms/forward I first saw at this ubatch was the one-time cost of the ubatch's first graph — prepare plus kernel JIT for the new shapes, 4.5 s above the following graphs — not a per-forward path.

Summary of what item 1 asked for, for every model that fits on this box:

	ub 512	ub 2048
host per prefill forward outside the device wait	0.03 ms (0.01 %)	~15 ms (1.3 %)
graph build (llama.cpp side)	1.5 ms	~2 ms
program-cache lookup	4 µs	0.18 ms
host copies (bind + staging)	7 µs	25 µs
GET_ROWS fallback	absent, one graph, no split	same
mutex/wait spin	inside `replay.wait`, which is the device leg	same

That is as far as item 1 goes on this box. Items 2 and 3 both need the 87 GiB Flash-Next file: 2 has no host item left to fix, and 3 is a device-side defect that only exists in that model's graph. One decision unblocks both — stop the owner's 27B NPU job, or accept UD-IQ3_XXS (76.4 GiB vs 75-82 GiB available) — and the run is one command each now that the profiler and the libhrx dispatch bundle are both in place.

PRs, neither merged: #646 (docs/hrx.md) and 1bit-MONSTER/llama.cpp#154 (the profiler, twelve phases, off by default).

docs: Flash-Next measured - the ~1.0 s is CPU-side, not the HRX host …

docs: the ~1.18 s is one GET_ROWS, declined by kMaxGetRowsRowCount (e…

Item 1 is closed, on the file itself, and item 2 has a one-line target. The owner's NPU job exited at 04:41 (114 GiB free), so I loaded the 87.3 GiB UD-IQ4_XS and measured it.

Qwen3.8-Flash-Next UD-IQ4_XS, -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0
  llama-bench                      1,795.4 ms/forward = 285.2 tok/s   (in-tree row: 286.63)
  HRX graph computes per forward   2 graphs: 2,566 dispatches in 618.6 ms, and a 1-dispatch one in 0.2 ms
  host phases inside HRX           <= 0.13 ms total (lookup, bind, cache key, transient, rebind,
                                   upload, launch, flush, download)
  libhrx dispatch bundle           2,567 dispatches/forward, 1,353.5 ms of device kernels per 2 forwards
                                   = 676.8 ms/forward = 37.7 %
  llama-server -lv 4 placement     6060 nodes, 1 off HRX (GET_ROWS x 1), 3 graph splits

So: 676.8 ms device (37.7 %), 0.13 ms of HRX host path, and ~1,177 ms (65.6 %) in one CPU node. The ~1.0 s is real; it is GET_ROWS fallback, the last item on this issue's own suspect list, and it is a single declined dispatch:

the node reads per_layer_token_embd.weight, which this file carries as IQ3_S [160, 320001536] — a ~20 GB table (token_embd.weight, the ordinary one, is Q8_0 [2560, 248320]);
ggml/src/ggml-hrx/dispatch_registration/common/dispatch-get-rows.cpp declines it at is_supported_row_count(): kMaxGetRowsRowCount = 262208 (line 23) against 320,001,536 rows;
the CPU then gathers 512 rows out of 20 GB per forward, and the cost is the page faults of touching it: ~35 ms at -p 128 (565.5 ms/forward) vs ~1.18 s at -p 512. Super-linear in tokens, because more tokens means more distinct rows and more cold pages. The graph splits 3 ways around it (HRX → CPU → HRX), so the device and host legs cannot overlap.

Item 2 is therefore: let that gather stay on HRX — raise/relax kMaxGetRowsRowCount for a table this shape, or add a dispatch for a gather from a table this large. Ceiling if it lands: the forward goes from 1,795.4 ms to ~677 ms of device plus whatever the gather costs on HRX, i.e. up to ~2.3x pp512 on this file — which is the ~2.4x roof in the brief, arrived at from the other end.

A correction to the tree while I was there: docs/hrx.md's "1,355 ms of the 1,786 ms pass (76 %)" is a two-forwards device total measured against a one-forward pass. 1,353.5 ms is exactly 2 × 676.8 ms from the bundle I measured. Per forward the device leg is 37.7 %, so the brief's 41.9 % was the closer figure and the published 76 % should come out. Fixed in #646.

Controls (a graph with no off-HRX node has no host leg at all): Qwen3-Coder-30B-A3B Q4_K_M 263.5 ms/forward with 0.02 ms of host phases and 1,161 dispatches; Qwen3.8-27B UD-Q4_K_XL 870.2 ms with 0.01 ms and 1,519. Both stay ~100 % device. rocprofv3 collects nothing on HRX (it exits 0 and writes no traceback files — HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets), so the device leg has to come from HRX_PROFILE_FILE + iree-profile statistics.

PRs, neither merged: #646 (docs: the whole profile, the correction, the GET_ROWS target) and 1bit-MONSTER/llama.cpp#154 (the host-path profiler, twelve phases, off by default).

docs: put the Flash-Next prefill forward on a gfx1151 roofline (engine …

The Flash-Next prefill forward on a gfx1151 roofline (TraceLens at ff705ed; no GPU run needed for the projection).

Method: #1175's TraceLens_model_roofline_projection parses a HF checkpoint, and this model exists here only as GGUF — qwen4exp is a hybrid llama.cpp architecture with no HF counterpart. So I used the same machinery a level down: TraceLens.PerfModel's GEMM / flash_attention formulas over shapes and exact per-tensor bytes read from the GGUF directory, against the peaks TraceLens keeps for gfx1151 after #1172 removed the bundled Radeon_8060S.json (measured spec already on this box: 256 GB/s, 59.39 TFLOP/s fp16, 118.78 TFLOP/s int4). Measured device time per class is from the libhrx dispatch bundle of the same model, so each row is roof against what the device actually did.

class	weights GB	touched GB	FLOPs T	mem roof ms	compute roof ms	measured ms
MoE experts	59.52	59.52	2.416	232.5	20.3	306.1
gated-delta-net layers (36)	2.25	2.25	2.126	8.8	35.8	33.7
token embedding + LM head	1.20	1.20	0.651	4.7	11.0	(dense)
full-attention layers (12)	1.02	1.02	0.671	4.0	11.3	136.3
MoE shared expert + router	0.50	0.50	0.306	2.0	5.2	(dense)
hyper-connections	0.34	0.34	0.644	1.3	10.8	35.1
PLE table	28.80	0.002	0	0.0	0.0	0.5
total	93.67	64.87	6.81	253	115	676.8

touched < weights because at 512 tokens with top-10 of 512 experts the expected number of untouched experts per layer is 512·e⁻¹⁰ = 0.02 — the expert weights are compulsory traffic — while the 28.8 GB PLE table is only gathered, one 90-byte IQ4_NL row per token per layer (2.2 MB). So the 87.3 GiB file does not put a 366 ms floor under a 512-token pass: 253 ms is the floor.

Three consequences:

The forward is memory-bound by 2.2× (253 ms against a 115 ms compute roof). No compute-side kernel rewrite can beat 253 ms on this file, which is worth knowing before anyone chases the expert GEMM's tile width again.
The device leg is at 37 % of its own compulsory-traffic roof — 2.7× the floor. The MoE expert GEMMs are the class closest to roof (306.1 ms against 232.5 ms, 76 %), so the slack is not there: 1,189 elementwise/copy/norm dispatches cost 106.6 ms with no weight traffic at all (~90 µs each, dispatch-bound), 273 f32 small matmuls cost 58.4 ms, and 395 dense projection GEMMs 136.3 ms.
The host leg is the largest single line in the forward: the one CPU GET_ROWS costs ~1,177 ms = 4.6× the memory roof of the entire pass, to move 2.2 MB. It is page faults on a cold 28.8 GB mmap'd table, not arithmetic — which is why it does not warm up between forwards (1,788.6 then 1,802.1 ms), while the 77 device-side get_rows dispatches cost 0.5 ms.

So item 2's ceiling, in roofline terms: give that gather a dispatch and the forward becomes ~620 ms of device, i.e. ~2.9× pp512 (285 → ~830 tok/s), and the file then sits at 37 % of its memory roof. The remaining gap to the floor is the dispatch-bound small ops — item 3's f32 batching is 58.4 ms of it and the elementwise/copy stream 106.6 ms.

Reproduce: ~/lb/i572/roofline_fn.py <gguf-prefix> <iree-profile statistics --format=jsonl> 512. The same table is in #646.

docs: where a Flash-Next prefill forward's 1.8 s goes, and the ~1.0 s…

README: current facts in the first heading and paragraph, badges; doc…

README: current facts, badges and links; docs: no line starts with an…

Bump HRX: llama.cpp 5e6388a5a (host-gap profiler, KV zero padding, De…

Status: item 1 shipped, item 2 merged-but-unproven, item 3 is the same file's graph. Both remaining measurements are now blocked on #669, and I have stopped running the 87.3 GiB file on HRX.

What shipped. The host-path profiler (llama.cpp #154) and the GET_ROWS row bound (#159) are both merged; the docs section in docs/hrx.md (#646) is merged. Fork 1bit/hrx-vulkan-patched is now at 5e6388a5a852281c6929303c1219665b2a6d4c0a, and the next HRX pin bump moves the engine's third_party/llama.cpp from d8d2f5b to it, carrying six commits: fe04824 (profiler), b8861a9 (#155 KV padding), 9e8453a/f61f290/01d7df4 (#156/#157/#158 dsv41) and 5e6388a (#159, this issue's item 2). Note for whoever raises it: tools/check_pins.py gates only the hrx-system pin against config/hrx-long-prompt-validated-pin.config, and that pin does not move in this bump — so this llama.cpp-only bump does not need the long-prompt probe re-run.

Item 2 numbers, partial and honest. #159 alone is not enough, which is what the placement analysis predicted:

the bound was necessary (the kernel's own index.assume capped ids at 262,207, mirrored by kMaxGetRowsRowCount), and it is now 2^29 with the byte-base proof;
but placement is two other policies: llama-model.cpp keeps every input-layer tensor on the CPU, and LLAMA_LAZY_MODE_AUTO mmaps every tensor over 4 GiB (qwen4exp.cpp asks for TENSOR_READ_LAZY on this one), which also short-circuits -ot;
measured on the pin-bump head (engine main + llama.cpp 5e6388a), pp512 ub512, default flags, 3 repeats: 281.9 tok/s against 285.2 tok/s on the cand-next build — no measurable change, as predicted;
with -lzm off -ot 'per_layer_token_embd\.weight=HRX0' the table does move: CPU_Mapped 644.14 MiB | HRX0 88688.02 MiB (61,222.07 + 27,465.95). That configuration is exactly what HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 forbids on this box: HRX pins the host weight mmap while holding 88.7 GB of device buffers, MemAvailable went to ~0 with no error from the runner, and the owner killed the run. I am not running that file on HRX again until HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 lands.

Item 3 is parked for the same reason, and I checked it rather than assuming. From the dispatch bundles already on disk: on Qwen3.8-27B UD-Q4_K_XL there is no ggml_mul_mat_f32_f32_wmma row at all — the f32 matmuls are fused (common.mul_mat.f32_f32_wmma (fused) in its matcher census), with only hrx_dp_f32_to_f16 (246 dispatches, 10.9 ms over 2 forwards) as the f32 traffic. On Qwen3-Coder-30B-A3B the router projection is already batched (qwen3_moe_router_projection_f32_four_row_wave32, 16 dispatches/forward). So Flash-Next's 273 standalone f32 dispatches / 58.4 ms are its own hc_*_inject (N=4) and ssm_alpha/beta shapes, and there is no runnable file on this box that exercises that pattern — a fix could be written but not validated, which the lane rules do not allow.

So: item 2 needs #669 (or the owner freeing the box for one long run with the table on the CPU and the gather dispatched — not viable at 88.7 GB), and item 3 needs item 2's file. I would rather leave both recorded than publish numbers from another thrash event.

Bump HRX: llama.cpp 5e6388a5a + hrx-system 82804197bcbe (#672)

Status after the pin bump (#694 3cddcd6, fork 4a58da9b0): the GET_ROWS bound is now on main, and it is
necessary but not sufficient for this issue — not closing.

What the pin carries (fork #160 4a58da9b0): dispatch-get-rows.cpp: kMaxGetRowsRowCount 2^29 → 2^30
(536,870,912 → 1,073,741,824) plus the two index.assume guards in get_rows_f32.loom. That is the DeepSeek-V4.1
Engram table (768,022,850 rows). This issue's table is per_layer_token_embd.weight at 320,001,536 rows — already
inside 2^29 after #159, so the bound was never this file's blocker.

What is still the blocker, at the pinned fork (4a58da9b0):

src/llama-model.cpp:1365   // there is very little benefit to offloading the input layer, so always keep it on the CPU
src/llama-model.cpp:1366   pimpl->dev_input = { cpu_dev, &pimpl->cpu_buft_list };
src/llama-model.cpp:2533   /*.lazy_mode =*/ LLAMA_LAZY_MODE_AUTO,

i.e. the two placement policies the 2026-10-30 comment named are unchanged: the input layer is pinned to the CPU,
and the lazy-mmap default short-circuits -ot. The lane's own measurement on the pin-bump head agreed — 281.9
tok/s default flags against 285.2 before, no change — and the configuration that does move the table
(-lzm off -ot 'per_layer_token_embd\.weight=HRX0') is exactly what #669 forbids on this box today (88.7 GB of
device buffers plus the pinned host mmap; MemAvailable went to ~0 and the owner killed the run).

What closes this issue: #669 landing (or the placement policies above changing so the gather goes to the device
without -ot), then the one row the lane already scripted — -lzm off -ot … with the gather dispatched — measured
against 285.2 tok/s and PPL 1.8511. Items 1 (host gap = 0.13 ms) and the roofline table stand as recorded.

Navigation Menu

Repository navigation

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572

Description

Activity

bong-water-water-bong commented on Oct 28, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions