Add chatLLaMa script by j3k0 · Pull Request #198 · ggml-org/llama.cpp

j3k0 · 2023-03-16T07:59:02Z

I'm not sure if this has a place in the repository. I did a bit of prompt engineering to get a conversation going with LLaMa, this is the script I use, which can serve as an example and "quickstart" to new users.

pjq · 2023-03-16T10:34:07Z

@j3k0 Nice script for new user like me.
Just curious, with your chat history context, will LLaMa continue the conversation?

j3k0 · 2023-03-16T11:02:03Z

In very rare instances, LLaMa will decide to end the conversation. It was occurring way more often before I told it it's a "never ending conversation" and a "10000 pages year long dialog".

I'm quite happy, even impressed with the result. Note that the examples I gave it (cat, Moscow) are excerpts from Wikipedia, I found that it reinforced LLaMa to use similar content as a source and tone for the answers, instead of the reddit/forums tone it had a tendency to produce.

pjq · 2023-03-16T15:02:24Z

Very nice, I am using your script to avoid it auto end the conversation.

D0han · 2023-03-16T23:17:38Z

This script could benefit from static analysis, please use https://www.shellcheck.net/ to make it more robust

G2G2G2G · 2023-03-17T05:19:58Z

the entirety of it is shared below.

is there trickery here? I'd (as a human) see this and expect not to reply / talk at all anymore and I am reading a past experience?

I assume you had to add this to get it to keep staying on or something?

j3k0 · 2023-03-17T06:33:07Z

@G2G2G2G The language model just tries to continue the text and make it self consistent. If you start with "here are 10 step to do XYZ. Step 1, do X. Step 2", then it will auto-complete it until it generated those 10 steps.

In this script, insisting in different ways that what comes below is a very long dialog, that never ends (etc.), reduces the likelihood that the auto-completion decides that the dialog is over ([end of text]). This still happens, but way less.

At that point it's just dark magic, I didn't do any statistical analysis or whatever to find the best prompt, just solving issues I experience with trial and error.

G2G2G2G · 2023-03-17T06:42:36Z

Well I think I was clear that I did understand that. But the specific text I outlined seems to suggest it's already over and to end it lol

anyway I thought issue 71 is the main reason stuff ends early.
(Your script doesn't help there, it still exits right at the same amount of text cuz of the token count max =[ )

j3k0 · 2023-03-17T06:48:23Z

@D0han done

j3k0 · 2023-03-17T06:55:22Z

@G2G2G2G You can increase --n_predict and get longer output, but it will end when out of state memory (can be increased with ctx_size). However I think issue 71 refers to what I'm trying to prevent here: the interaction is often ended by the model before reaching any limits (with this special [end of text] token). I played around in main.cpp to prevent it and force-insert reverse_prompt tokens when end of text is generated by the model, however the models internal state becomes inconsistent at that stage (it forgot what it was doing), so there is no point.

mdonaberger · 2023-03-17T22:47:29Z

Great script, thank you! I had been trying to build my own chat script and your prompt seems to be the key!

G2G2G2G · 2023-03-18T02:05:49Z

@j3k0 ah I see, thanks for the input.

inconsistent at that stage (it forgot what it was doing), so there is no point.

wow maybe it has dementia =[

thement · 2023-03-18T18:26:10Z

How about making a directory named contrib and put the script there? It could be a place to put other related and useful tools.

Green-Sky · 2023-03-18T19:02:14Z

How about making a directory named contrib and put the script there? It could be a place to put other related and useful tools.

Good Idea. but let's keep it consistent with whisper.cpp. There is the examples directory.

https://github.com/ggerganov/whisper.cpp/tree/master/examples

rabidcopy · 2023-03-19T14:36:54Z

May I suggest increasing n_predict to 2048? From my understanding having -n 1024 more or less limits the context size to that number and chat sessions will cut off sooner basically. #266 (comment) Again I may be wrong but I do know setting -n 2048 allows for effectively twice the session length before it reaches a hard stop. I'll try to do more testing on this. If I lower it to -n 96 I can get only 3 dog facts listed off before it comes to a hard stop. Regardless of how high -c is.

Ref d7def1a

j3k0 · 2023-03-19T16:30:48Z

May I suggest increasing n_predict to 2048?

Done and rebased my branch onto master.

README.md - Update notable forks section

* Add chatLLaMa script * Fix shellcheck errors and do some cleanup * Move chatLLaMa script to `examples` directory * Reduce chatLLaMa context size to 2048 Ref d7def1a * Include n_predict to 2048 in examples/chatLLaMa

…#198) * Load all MoE experts during warmup Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * Unify warmup to one token --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

* Add chatLLaMa script * Fix shellcheck errors and do some cleanup * Move chatLLaMa script to `examples` directory * Reduce chatLLaMa context size to 2048 Ref 42c481e * Include n_predict to 2048 in examples/chatLLaMa

…org#241) (ggml-org#271) The Windows CUDA x64, Linux CUDA and Windows HIP release jobs build with GGML_NATIVE=OFF but without GGML_BACKEND_DL/GGML_CPU_ALL_VARIANTS. With native detection off and no explicit ISA flags, ggml enables AVX/AVX2/FMA/F16C by default, so each bundle ships a single AVX2-only ggml-cpu library. On a CPU without AVX2 the process dies at startup with no message: even `llama-server --version` exits silently (ggml-org#241). The CPU and Vulkan bundles already build every variant and pick one at runtime (Vulkan since ggml-org#198). Build those three bundles with GGML_BACKEND_DL=ON and GGML_CPU_ALL_VARIANTS=ON as well. The arm64 CUDA leg already used BACKEND_DL with GGML_CPU=OFF and is unchanged. Claude-Session: https://claude.ai/code/session_012Q6ecxozUXFFFLE3N2SMiR Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 9a4dd89)

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com>

j3k0 force-pushed the chatLLaMa branch from 6d4cab1 to a81341d Compare March 17, 2023 07:12

gjmulder added the enhancement New feature or request label Mar 17, 2023

j3k0 added 5 commits March 19, 2023 18:29

Add chatLLaMa script

fdb864a

Fix shellcheck errors and do some cleanup

2aaf379

Move chatLLaMa script to examples directory

b6bcd01

Reduce chatLLaMa context size to 2048

b8c383a

Ref d7def1a

Include n_predict to 2048 in examples/chatLLaMa

1b8f8ad

j3k0 force-pushed the chatLLaMa branch from e587f7c to 1b8f8ad Compare March 19, 2023 16:29

ggerganov approved these changes Mar 21, 2023

View reviewed changes

ggerganov merged commit 3ab3e65 into ggml-org:master Mar 21, 2023

rooprob pushed a commit to rooprob/llama.cpp that referenced this pull request Aug 2, 2023

Merge pull request ggml-org#198 from trholding/notable-forks-patch

c1a0c6e

README.md - Update notable forks section

Conversation

j3k0 commented Mar 16, 2023

Uh oh!

pjq commented Mar 16, 2023

Uh oh!

j3k0 commented Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pjq commented Mar 16, 2023

Uh oh!

D0han commented Mar 16, 2023

Uh oh!

G2G2G2G commented Mar 17, 2023

Uh oh!

j3k0 commented Mar 17, 2023

Uh oh!

G2G2G2G commented Mar 17, 2023

Uh oh!

j3k0 commented Mar 17, 2023

Uh oh!

j3k0 commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdonaberger commented Mar 17, 2023

Uh oh!

G2G2G2G commented Mar 18, 2023

Uh oh!

thement commented Mar 18, 2023

Uh oh!

Green-Sky commented Mar 18, 2023

Uh oh!

rabidcopy commented Mar 19, 2023

Uh oh!

j3k0 commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

j3k0 commented Mar 16, 2023 •

edited

Loading

j3k0 commented Mar 17, 2023 •

edited

Loading