Repository navigation
Add parameter to ignore end of text token - #181
Merged
Merged
Conversation
ggerganov
approved these changes
Mar 19, 2023
Seunghhon
pushed a commit
to Seunghhon/llama.cpp
that referenced
this pull request
Apr 26, 2026
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* Adding gp option to llama-bench Similar to pg, but it only looks at TG speed with a given prompt length. * Make q8_0_r4 work with tensor row sizes that are not a multiple of 128 They still need to be divisible by 32. * Make q8_0_r4 work with tensor row sizes that are not a multiple of 128 .. on NEON * Make q8_0_r4 work with tensor row sizes that are not a multiple of 128 .., on AVX2 * Make q4_0_r4 work with tensor row sizes that are not a multiple of 128 .., on AVX2 * Make q4_0_r4 work with tensor row sizes that are not a multiple of 128 ... on NEON * Make q4_0_r4 work with tensor row sizes that are not a multiple of 128 ... on Zen4. Also fix q8_0 K-cache for head sizes that are not multiple of 128. --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* Slightly faster AVX2 implementation for q4_k_r4 * Even better AVX2 implementation for q4_k_r4 We now arrive at PP-512 = 328 t/s for LLaMA-3.1-8B on a Ryzen-5975WX CPU, up from 291 t/s when I last measured on 3c5f872. With FA and Q8_0 K-cache we get to 339.5 t/s. * Fix llama-bench labels that I broke with ggml-org#181 * Faster AVX2 implementation for q5_k_q4 We arrive at 302 t/s for LLaMA-3.1-8B on a Ryzen-5975WX CPU, up from 273 t/s. * Use AVX2 implementation of q4_k_r4 and q5_k_r4 also on Zen4 After the changes I made to AVX2, it ends up being slightly faster compared to what I had for Zen4. * Minor tweak * Cleanup --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
ljubomirj
pushed a commit
to ljubomirj/llama.cpp
that referenced
this pull request
May 6, 2026
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 23, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 24, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com>
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
hjotha
pushed a commit
to hjotha/beellama.cpp
that referenced
this pull request
Oct 25, 2026
…-state kvarn: persist exactly the F16-only stage groups in self-contained state
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the --ignore-eos switch which prevents generation of the end of text (eos) token. This can be useful to avoid unexpected terminations in interactive mode and to force the model to generate longer output.
This is implemented by setting the logits of the eos token to zero, which seems to work well enough, but I am not sure if there may be any unwanted side effects.