Repository navigation
HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture - #28935
Merged
IMbackK merged 1 commit intoSep 16, 2026
Merged
Conversation
It's found the MoE ncols_opt tile heuristic needs to be broadened
to include the RDNA3.5 architecture.
The code change is implemented in ggml/src/ggml-cuda/mmq.cu
and just change the GGML_CUDA_CC_IS_RDNA3_0 to
GGML_CUDA_CC_IS_RDNA3 in the condition.
The dense dispatch logic remains unchanged.
The Test machine configuration we used is
AMD Radeon 8060S, gfx1151 (RDNA3.5), 20 CU, wave32
+ AMD Ryzen AI MAX+ 388, 8C/16T, 23.79 GB RAM
we complete the Correctness verification and performance evaluation as follows:
test-backend-ops test -b ROCm0 -o MUL_MAT -p type_a=<q4_K|q5_K|q4_0|q5_0>
test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a=<q4_K|q5_K|q4_0|q5_0>
all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14;
MUL_MAT_ID 84/84, 3/3, 74/74, 3/3
Performance result on target machine:
LFM2.5-8B-A1B-UD-Q4_K_M (Q4_K MoE) +16.198% [+12.704, +19.799] 8/8
Qwen1.5-MoE-A2.7B-Q2_K (Q2_K MoE) +6.189% [ +5.245, +7.141] 8/8
pooled (16 pairs) +11.081% [ +7.972, +14.279] 16/16
Token generation (tg128) is unchanged on the Q4_K MoE model and +2.188%
[+0.905, +3.488] on the Q2_K one.
github-actions
Bot
added
ggml
|
Hi @WenqiangJia2026, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
Contributor
Author
The actual AI usage has been disclosed, and the commit message and PR description |
Contributor
|
Please benchmark the performance for all relevant combinations of physical batch sizes and model types and post the results you get with |
Contributor
Author
Hi I have completed the experiments you mentioned before, below are the details and results: the build command: the quantize part command: curl -L -o wikitext-2-raw-v1.zip llama-imatrix -m granite-3.0-3b-a800m-instruct-f16.gguf llama-quantize --imatrix granite-3.0-3b-a800m-instruct.imatrix.gguf the performance command: llama-bench -m granite-3.0-3b-a800m-instruct-.gguf scripts/compare-llama-bench.py -t llama-bench -i sweep.jsonl -b 4c9233c -c b3ea2b2 the final result as below:
summarized the results as below: Same data expressed as percent change, the compute details as below:
Across the 200 (type, ubatch) cells: 77 significantly faster, 8 significantly and there are 8 statistically significant slower as below: the ub=32 case I found the result change significantly(IQ4_NL 1.41 vs IQ2_XS 0.93), the original data was attached, which help to reproduce above result. |
JohannesGaessler
approved these changes
Sep 16, 2026
IMbackK
approved these changes
Sep 16, 2026
quimmedes
pushed a commit
to quimmedes/cafe-llama.cpp
that referenced
this pull request
Sep 16, 2026
…ml-org#28935) It's found the MoE ncols_opt tile heuristic needs to be broadened to include the RDNA3.5 architecture. The code change is implemented in ggml/src/ggml-cuda/mmq.cu and just change the GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3 in the condition. The dense dispatch logic remains unchanged. The Test machine configuration we used is AMD Radeon 8060S, gfx1151 (RDNA3.5), 20 CU, wave32 + AMD Ryzen AI MAX+ 388, 8C/16T, 23.79 GB RAM we complete the Correctness verification and performance evaluation as follows: test-backend-ops test -b ROCm0 -o MUL_MAT -p type_a=<q4_K|q5_K|q4_0|q5_0> test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a=<q4_K|q5_K|q4_0|q5_0> all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14; MUL_MAT_ID 84/84, 3/3, 74/74, 3/3 Performance result on target machine: LFM2.5-8B-A1B-UD-Q4_K_M (Q4_K MoE) +16.198% [+12.704, +19.799] 8/8 Qwen1.5-MoE-A2.7B-Q2_K (Q2_K MoE) +6.189% [ +5.245, +7.141] 8/8 pooled (16 pairs) +11.081% [ +7.972, +14.279] 16/16 Token generation (tg128) is unchanged on the Q4_K MoE model and +2.188% [+0.905, +3.488] on the Q2_K one.
tungel
pushed a commit
to tungel/llama.cpp
that referenced
this pull request
Sep 18, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count.
tungel
pushed a commit
to tungel/llama.cpp
that referenced
this pull request
Sep 18, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count.
tungel
pushed a commit
to tungel/llama.cpp
that referenced
this pull request
Sep 18, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count.
tungel
pushed a commit
to tungel/llama.cpp
that referenced
this pull request
Sep 19, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count.
mbednarek360
pushed a commit
to mbednarek360/llama.cpp
that referenced
this pull request
Sep 22, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count.
1 task done
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Sep 26, 2026
…NA3.5 Upstream ggml-org#28552/ggml-org#28935 land the routed-MoE ncols picker on RDNA3.5, which Finding ggml-org#10 measured flat against our static J=48 cap, so the cap goes. Upstream ggml-org#28102 routes WMMA to MMA_F16 at D=256 and bypasses our tile config at prefill; guard it off on RDNA3.5 pending the G/N A/B. Assisted-by: Claude Opus 5.5
tungel
pushed a commit
to tungel/llama.cpp
that referenced
this pull request
Sep 27, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count. [fold target for the qwen4exp/HC/QSA/indexer beta group — see the follow-up commit folds] Folds beta/mmb-general: the deferred half of 0001 (MMB stand-down of the pair fusion) and 0027 (MMVQ routed band W=16 + gfx1151 dense odd-row band). The qwen4exp/HC/QSA/indexer group folds into block 15 (it needs block 15's context, which the beta series was authored against).
tebruno99
pushed a commit
to tebruno99/llama.cpp-rocm10-patches
that referenced
this pull request
Oct 7, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count. [fold target for the qwen4exp/HC/QSA/indexer beta group — see the follow-up commit folds] Folds beta/mmb-general: the deferred half of 0001 (MMB stand-down of the pair fusion) and 0027 (MMVQ routed band W=16 + gfx1151 dense odd-row band). The qwen4exp/HC/QSA/indexer group folds into block 15 (it needs block 15's context, which the beta series was authored against). Campaign fold (wip/moe-expert-cache, beta): `llm_arch_supports_sm_tensor()` rejects `LLM_ARCH_GEMMA4` until the segmented host-resident-expert async upload is finished -- gemma4's fused expert tensor (`ffn_gate_up_exps`) has a segmented split layout and the MMQ tail pad makes the range non-row-aligned, so the per-ubatch `-ncmoe` upload has no correct path. All offload routes are covered at the arch level; `-sm layer` is unaffected. r6 (2026-10-05, issue ggml-org#95): publish `GGML_USE_HIP` on the `ggml` target so the qwen4exp tensor-split gate sees it in dynamic-backend builds. `ggml_add_backend()` only adds `GGML_USE_<backend>` to `ggml` when `GGML_BACKEND_DL` is OFF, and the published containers build with `-DGGML_BACKEND_DL=ON` (needed for `GGML_CPU_ALL_VARIANTS`), so `src/llama-arch.cpp`'s `#ifdef GGML_USE_HIP` compiled to the HIP-absent branch and `-sm tensor` for qwen4exp failed with "not implemented for architecture 'qwen4exp'" even though the HIP backend was built. The static build already got the macro from `ggml_add_backend(HIP)`; publishing it in `ggml/src/ggml-hip/CMakeLists.txt` covers both. Issue ggml-org#95 (@ethanjjjjjjj). Folded 2026-10-07 (PR ggml-org#106, @briansp2020; issue ggml-org#105): `LLAMA_KV_N_PAD_MIN` raises the n_kv padding floor (opt-in; the default 256 is unchanged), keeping the per-device tensor-split graphs stable for longer; and the conv-state tail is copied straight into each rollback slot (one 2D memcpy on the strided view, about 144 fewer kernels per verify graph; `LLAMA_CONV_TAIL_CONT=1` restores the cont+copy).
stew675
added a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 8, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count. [fold target for the qwen4exp/HC/QSA/indexer beta group — see the follow-up commit folds] Folds beta/mmb-general: the deferred half of 0001 (MMB stand-down of the pair fusion) and 0027 (MMVQ routed band W=16 + gfx1151 dense odd-row band). The qwen4exp/HC/QSA/indexer group folds into block 15 (it needs block 15's context, which the beta series was authored against). Campaign fold (wip/moe-expert-cache, beta): `llm_arch_supports_sm_tensor()` rejects `LLM_ARCH_GEMMA4` until the segmented host-resident-expert async upload is finished -- gemma4's fused expert tensor (`ffn_gate_up_exps`) has a segmented split layout and the MMQ tail pad makes the range non-row-aligned, so the per-ubatch `-ncmoe` upload has no correct path. All offload routes are covered at the arch level; `-sm layer` is unaffected. r6 (2026-10-05, issue ggml-org#95): publish `GGML_USE_HIP` on the `ggml` target so the qwen4exp tensor-split gate sees it in dynamic-backend builds. `ggml_add_backend()` only adds `GGML_USE_<backend>` to `ggml` when `GGML_BACKEND_DL` is OFF, and the published containers build with `-DGGML_BACKEND_DL=ON` (needed for `GGML_CPU_ALL_VARIANTS`), so `src/llama-arch.cpp`'s `#ifdef GGML_USE_HIP` compiled to the HIP-absent branch and `-sm tensor` for qwen4exp failed with "not implemented for architecture 'qwen4exp'" even though the HIP backend was built. The static build already got the macro from `ggml_add_backend(HIP)`; publishing it in `ggml/src/ggml-hip/CMakeLists.txt` covers both. Issue ggml-org#95 (@ethanjjjjjjj). Folded 2026-10-07 (PR ggml-org#106, @briansp2020; issue ggml-org#105): `LLAMA_KV_N_PAD_MIN` raises the n_kv padding floor (opt-in; the default 256 is unchanged), keeping the per-device tensor-split graphs stable for longer; and the conv-state tail is copied straight into each rollback slot (one 2D memcpy on the strided view, about 144 fewer kernels per verify graph; `LLAMA_CONV_TAIL_CONT=1` restores the cont+copy).
sjaccaud
added a commit
to sjaccaud/llama.cpp
that referenced
this pull request
Oct 27, 2026
qwen4exp (Qwen3.8-Flash-Next) support on the rdna-boosts core, squashed from the beta/qwen4exp delta (fork qwen4exp tip dd4301fb4, re-based to the 465e49b-era core; promoted to the delivery at the 050dde5 re-base 2026-09-07): - QSA layers: fused indexer top-k (GGML_OP_INDEXER_TOPK, radix), sparse flash attention (GGML_OP_FLASH_ATTN_QSA) -- the default FA path; LLAMA_QSA_SPARSE_FA=0 keeps dense; CPU reference for the sparse op - fused decode ops GGML_OP_HC_MIX / GGML_OP_HC_COMBINE (+ kernel geometry, rms/gamma fold, F32/Q8_0 inject fold, head-call fusion) - managed lazy reader (llama-lazy-reader.cpp/.h) with PLE n-gram row loading and batched cold-page fetch (LLAMA_LAZY_IO_THREADS) - MTP draft-head support for the Flash-Next GGUFs (--spec-type draft-mtp) - WS4 hyperconn prefill fusions (DSV4_HC_COMB/PRE/POST); ggml sched alloc-fallback sync fix; QSA dense shortcut (DEFAULT ON) + LLAMA_QSA_OFF - QSA decode campaign (2026-09-07): fused INDEXER_POOL/SCORE ops, f16 indexer caches, incremental derived block-vector cache, two-round 16-bit topk select, CUDA device-description gfx-id exposure - arch policy (2026-09-07 crossover tables): dense decode below the per-arch depth, QSA at/above (gfx1151 default 65536); LLAMA_QSA_DENSE_DECODE_UNTIL overrides - MoE weighted-reduction fusion on the Meta-TP path; concat-transposed, mmid-512x10, repeat-absorb hc-combine absorb; model support in src/models/qwen4exp.cpp Re-based onto the 050dde5 delivery core 2026-09-07: the only upstream-drift conflict (ggml-cuda/common.cuh) resolved by keeping both the upstream gfx90c GCN-APU arch macros and the block's exact-SKU GGML_CUDA_CC_IS_GFX1151 predicate. Upstream's branchless Q4_K/Q5_K mmvq unpack + L2 prefetch and the block's k-quant/decode additions coexist (mmqq/vecdotq regions disjoint). Full beta notes + validation: beta/qwen4exp/README.md. Amended 2026-09-08 with the compiler-warning cleanup for the ggml-clang / ggml-clang++ (system clang 16) and ROCm 7.14 HIP builds: drop the unused n_blocks local in ggml_indexer_fill (ggml.c); add the GGML_OP_INDEXER_SCORE/GGML_OP_INDEXER_FILL case labels to the CPU compute-forward switch (GPU-only fused ops; the CPU plan phase already aborts on them); drop the unreachable break after the noreturn GGML_ABORT in the HC_MIX/HC_COMBINE CPU type dispatches (ops.cpp); restore the GGML_CUDA_QSA_INDEXER_CACHE env tri-state (0/1/2) so the idx_cache == 2 debug probe is actually reachable (was narrowed to bool); size_t loop counter in the device gfx-id sniff. No generated-code or runtime-behavior change in default configs. Amended 2026-09-08 (2nd) with the qwen4exp tensor-split backend gate: llm_arch_supports_sm_tensor(qwen4exp) is now true only on HIP builds (#ifdef GGML_USE_HIP). The fused QSA/HC/WS4 ops tensor-split through the meta backend are validated on ROCm only (3x R9700); on backends that cannot run them on-device (Vulkan, Metal, SYCL) the CPU-fallback subgraphs leave the meta splitter unable to reconcile mirrored-vs-split operand states (abort at ggml-backend-meta.cpp handle_generic, e.g. the qwen4exp gated-attention MUL on Vulkan), and NVIDIA CUDA is untested, so those builds keep upstream's unsupported (clean error / arch-test SKIP) instead of crashing. Amended 2026-09-10: freed-cell KV handling moved from host-side zeroing to kernel-side masked-V elimination, and the earlier host zeroing (seq_rm/seq_keep/clear per-free memsets, 2026-09-09 fold of the strix aad5adb port) is REMOVED. The llama-kv-cache files are back to the upstream state (no zero_freed/rows_hw/sharers wiring; evicting a KV sequence is pure host cell bookkeeping again on every backend). In its place block 14 now carries the three kernel fixes that make freed-cell content unreadable in masked flash attention, so the host workaround is unnecessary: - HIP fattn-tile.cuh: in the packed-bf16 PV path, zero the per-warp V register copies of rows whose P is exactly +0.0 across the warp's columns (the fully-masked rows) before the bf16 dot. - HIP fattn-mma-f16.cuh: after each V-tile slice is staged in shared memory, zero the rows the mask tile marks blocked (-inf) for every query column of the block; one extra uniform barrier on the masked path (ncols2 > 1 || mask_h) only; compile-time excluded for the V_is_K_view and NVIDIA swizzled (swz_V) paths. - Vulkan flash_attn_cm1.comp: never read V of fully masked columns (per-column liveness -> V of dead columns stays +0.0); flash_attn.comp scalar path likewise skips the V load for dead columns. All three are unconditional in their kernel paths (no arch/env gating): they are correctness fixes for any masked flash-attention column whose cell is stale/freed (batch serving, KV eviction), active by default on every device. They were root-caused and validated on the Strix Halo box (single gfx1151): 16/16 identical-request determinism gates pass on both ROCm and Vulkan with the host zeroing disabled, bit-identical across every KV cache type each backend's FA supports (ROCm f16/bf16/q8_0/q4_0; Vulkan also q4_1/q5_0/q5_1/iq4_nl), with test-backend-ops FLASH_ATTN_EXT vs CPU 4591/4591 (ROCm) and 7822/7822 (Vulkan) and no measurable decode/prefill regression (tg128 within 0.05% at depth-16384; pp within single-run drift). Full record: wip/strix-halo/kvzero/RECORD-2026-09-09.md and wip/kv-sign-leak/HANDOVER-2026-09-09-mma-f16.md in the delivery repo. The 2026-09-09 amendment text below is retained as history only; the freed-cell zeroing gate it describes no longer exists in the block. Amended 2026-09-12 (seventh) with the MTP-export logits-purity fix: an unmasked embeddings_nextn export (the draft-mtp target) used to defer the last layer's output-row gather, so the last layer's ffn tail ran on the full ubatch and the prefill's last-position logits shifted by a ULP against --spec-type none. The last layer now always gathers the output rows before its tail (exactly the plain path) and builds a second full-row tail solely for t_h_nextn when the chunk drops rows (n_outputs < n_tokens); a decode/verify batch drops none, so nothing is duplicated there. Verified on gfx1151: the mstep NEXTN=1 prefill mismatch is gone (0 mismatches; was 1 at pos = 4293), the W=4 width probe is 0 mismatches with the W=8 position list unmoved, the default and forced-sparse text gates are byte-identical, MTP acceptance is bit-identical (f16 0.51678 = 77/149 pre- and post-fix), and the graph is a no-op on every non-NEXTN path. Instrument: wip/strix-halo/qsa-item4/. Amended 2026-09-12 (eighth) with the decode/verify band-uniformity fix for the QSA indexer score. The score's matmul carries the indexer heads in its N dimension, so its ne11 is n_idx_h * n_tps (4 * n_tps for qwen4exp); block 08's "keep the verify batch on the decode kernel" guard only covered ne11 <= MMVF_MAX_BATCH_SIZE (8), so from n_tps = 3 the verify batch fell through to MMF while decode (n_tps = 1) stayed on MMVF. The two families accumulate the truncated dot product differently, so the indexer score differed by a ULP and flipped a top-k near-tie: the forward was bit-identical to decode for 101 steps and then diverged. The guard now keeps the whole flattened band on the decode family (ne11 <= MMVF_MAX_BATCH_SIZE_FLAT = 32) and mul_mat_vec_f is instantiated for ncols_dst 9..32. The guard is block 08's; block 14 extends it because it is the block that introduces the flattened batch. Verified on gfx1201 (3x R9700, layer split): mstep W = 1,2,3,4,5,8 all 0 mismatches (was W >= 3 impure), the W = 1 reference Thash is unchanged (2bd73063dd0a9524) so decode numerics are untouched, the f16 band stays pure, and the forced-sparse q8_0 and default text gates are byte-identical (a4cdc10dfb6c / 2e078b6966c0). Cross-check on gfx1151 pending. Instruments: wip/strix-halo/qsa-item4/. Amended 2026-09-17 (re-base onto ebbb185): the pair-fusion MUL_MAT_ID arm's ncols_opt arch gate is broadened from GGML_CUDA_CC_IS_RDNA3_0 to GGML_CUDA_CC_IS_RDNA3, matching upstream ggml-org#28935 which broadened the standalone ggml_cuda_mul_mat_q tile heuristic to RDNA3.5 (gfx1151). No-op on RDNA3_0 and RDNA4 (the predicate is unchanged there), but it keeps the fused gate+up pair's tile selection consistent with the standalone on gfx1151 instead of falling back to the full token count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
the code is add code to broaden the support for rdna 3.5 arch,
the change for correction and performance improvement has verified on local target.
Additional information
the command for build:
cmake -S . -B build -G Ninja -DCMAKE_C_COMPILER=<rocm>/bin/clang.exe -DCMAKE_CXX_COMPILER=<rocm>/bin/clang++.exe -DCMAKE_HIP_COMPILER=<rocm>/bin/clang++.exe -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DGGML_HIP_NO_VMM=ON -DGGML_CUDA_FA_ALL_KERNELS=ON -DCMAKE_BUILD_TYPE=Releasethe local target:
CPU(AMD Ryzen AI MAX+ 388,8C/16T,23.79 GB RAM)
GPU (AMD Radeon 8060S,gfx1151 (RDNA3.5),20 CU,wave32)
toolchain (ROCm 7.2,HIP 7.2.60201-38d754472,clang 21.0.0git,CMake 3.27.2,Ninja 1.11.0)
we verify the Correctness and performance result use below command:
test-backend-ops test -b ROCm0 -o MUL_MAT -p type_a=q4_K / q5_K / q4_0 / q5_0test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a=q4_K / q5_K / q4_0 / q5_0llama-bench -m <model> -p 2048 -n 128 -b 2048 -ub 512 -ngl 99 -fa 1 -lm none -r 3 -o jsonthe final result as below:
LFM2.5-8B-A1B-UD-Q4_K_M (Q4_K MoE) pp2048 +16.198% CI [+12.704, +19.799] 8/8
Qwen1.5-MoE-A2.7B-Q2_K (Q2_K MoE) pp2048 +6.189% CI [ +5.245, +7.141] 8/8
all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14; MUL_MAT_ID 84/84, 3/3, 74/74, 3/3
This is verified on only one gfx1151 device.
Requirements