Repository navigation
Flash attention (CUDA/HIP): garbage output at long context on gfx1201 (RDNA4) - immediate EOS, head_dim 256 #30884
Description
With flash attention enabled, long-context prefill produces corrupted logits on gfx1201 (AMD Radeon AI R9700, RDNA4) for a model with head_dim 256 (Qwen3.8-27B). The first generated token is EOS with high confidence (generation stops immediately, no decode step runs), so the corruption is in the prefill itself. The affected context lengths form a non-monotone band: ~66.5k and ~87.5k tokens are garbage while ~76.5k is clean.
Reproduced on the official upstream build b11755 (commit 704366d) and on a local HIP build of 1fc41fd68 - ggml/src/ggml-cuda/ is unchanged between these two commits (verified with git log 1fc41fd68..704366d76 -- ggml/src/ggml-cuda/).
Environment
- llama.cpp b11755 (commit 704366d), official Windows x64 ROCm 10.0 prebuilt (
llama-b11755-bin-win-rocm-10.0-x64, Clang 20.1.8); also reproduced on a local HIP build of 1fc41fd68 - GPU: AMD Radeon AI R9700 (gfx1201, RDNA4, 32 GB), ROCm 10.0, Windows x64, discrete GPU (no unified memory)
- Model:
Qwen3.8-27B-UD-Q4_K_M(unsloth GGUF), archqwen35: 65 blocks (64 main + 1 MTP), hybrid attention withfull_attention_interval=4(16 full-attention layers), n_head=24, n_kv_head=4 (GQA 6), head_dim=256, context_length=262144
Reproduction
Server (everything on GPU):
llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf -ngl 99 -c 100000 -b 2048 -ub 288 -fa on -ctk f16 -ctv f16 --port 18080 --no-ui --parallel 1 --fit off
Deterministic prompt: N filler paragraphs with a codename embedded in the middle, then a question. N=877 gives 87,472 prompt tokens; N=665 gives 66,326; N=767 gives 76,502. Request: POST /completion with {"n_predict": 300, "temp": 0, "logprobs": 10}.
Generator (Python, verbatim):
names = ['Kestrel','Lighthouse','Beacon','Compass','Harbor','Anchor','Meridian','Pinnacle','Cascade','Summit','Vanguard','Sentinel','Pioneer','Horizon','Latitude','Altitude','Vector','Bearing','Gyro','Astrolabe','Chrono','Tide','Fathom','Sonar','Drift']
places = ['Portland','Seattle','Tacoma','Eugene','Boise','Spokane','Bellingham','Walla Walla','Astoria','Hood River','Salem','Corvallis','Bainbridge','Vashon','Cedar Falls','Yakima','Wenatchee','Olympia','Anacortes','Friday Harbor','Coos Bay','Newport','Astoria','Longview','Kelso']
N = 877 # also: 665, 767
mid = N // 2
paras = []
for i in range(1, N + 1):
n1 = names[(i * 3) % 25]; n2 = names[(i * 7 + 3) % 25]
pl = places[(i * 5 + 1) % 25]; pl2 = places[(i * 11 + 4) % 25]
y = 1987 + i
if i == mid:
paras.append(f"In the spring of {y} the board approved a confidential research initiative to build a low-cost atmospheric sensor network. The project was given an internal codename that the founding team chose to reflect a winter storm observed during the planning meeting. The project codename is AMBERFROST9317. Funding came from retained earnings and a small state science foundation grant, and the team was deliberately kept to fewer than forty people to protect the work from competitors.")
else:
paras.append(f"During {y} the cooperative introduced the {n1} series of field instruments, developed jointly with partners in {pl} and {pl2}. Early adopters praised the {n1} line for its rugged housing and simple calibration routine, while the {n2} program expanded into marine navigation and coastal surveying. Annual reports from that period describe steady growth in service contracts, a modest expansion of the engineering staff, and continued investment in predictive modeling tools that later became a core part of the company's software services division.")
prompt = "\n".join(paras) + "\n\nQuestion: What is the project codename mentioned in the passage? Reply with only the codename, nothing else."Observed behavior
For the affected lengths the first generated token is EOS: stop_type=eos, predicted_n=1, and no decode evals occur at all (the corrupted logits come from prefill). First-token top-10 logprobs from the 87,472-token run on b11755:
EOS: -0.202, "\n\n": -2.004, "": -3.047, "\n\n\n": -8.288, "\n": -8.589,
" Reply": -8.681, "\n\n\n\n": -9.725, " (": -9.862, " Just": -10.184, " What": -10.784
The correct answer (the codename) is not in the top 10 at all.
Band map (prompt tokens / FA-visible KV length, which the kernel sees padded to a multiple of 256):
| prompt tokens | FA-visible (nkv) | b11755 prebuilt (f16 KV) | 1fc41fd68 local build |
|---|---|---|---|
| 66,326 | 66,560 (260x256) | GARBAGE (immediate EOS) | GARBAGE (immediate EOS); with q4_0 KV: 115 tokens of off-task text (mild) |
| 76,502 | 76,544 (299x256) | GARBAGE (immediate EOS) | CLEAN - answers "AMBERFROST9317" correctly (15 tokens) |
| 87,472 | 87,552 (342x256) | GARBAGE (immediate EOS) | GARBAGE (immediate EOS) |
Results are deterministic: same binary + same prompt gives the same result.
Things that do not change the result (at 87,472 tokens, 1fc41fd68 build)
- KV cache type: f16, q4_0, q8_0 - all immediate EOS (66,326 tokens: f16 immediate EOS, q4_0 mild)
- context size: 100000 and 262144
- ubatch size: 288 and 512
FA off is not a viable option: without flash attention the prefill drops from ~950 tok/s to ~50 tok/s (the 87.5k test hit a 15-minute request timeout at 42% progress), and with quantized KV the server refuses to start ("quantized V cache requires flash_attn to be enabled").
Kernel-path notes
-
With
-b 2048 -ub 288the final partial batch ends in its own 4-token ubatch (e.g. for the 76,502-token prompt the tail is [288, 288, 146, 4]; for 87,472 it is [288x5, 12, 4]). The first sampled token is produced at the last prefill position, i.e. by that 4-token ubatch. -
The nkv seen by the FA kernel is padded to a multiple of 256 (this is where the "FA-visible" numbers in the band map come from).
-
On gfx1201 that nq=4 ubatch runs the TILE kernel (
fattn-tile.cuh), non-stream_k, with parallel_blocks=8 and aflash_attn_combine_resultsmerge; the wide prefill ubatches (nq=288) run the MMA_F16 kernel. -
Instrumented build (1fc41fd68 + launch-parameter logging) confirms the last prefill FA call at the 87,472-token length, once per each of the 16 full-attention layers:
[FA] nq=4 nkv=87552 nbatch_fa=32 ntiles_KV=2736 ntiles_dst=12 pb=8 sk=0 sp=0 DV=256 nc1=4 nc2=2 wp=4 mb=3 nsm=32 blk=(1,8,12) -
On this hardware all three FA kernels (MMA/TILE/VEC) accumulate the VK^T Q dot product in f16 (half2); the KQ logits, max, and row sum are f32.
-
Open question: is the corruption produced by the 4-token TILE ubatch, or do earlier wide MMA ubatches write corrupted K/V state?
Hypothesis ruled out: f16 softmax overflow
The kernels mitigate f16 overflow with FATTN_KQ_MAX_OFFSET in fattn-common.cuh (a 2^3 shift of the softmax max, added for #18606). I built a variant with the offset raised to 2^12: at 87,552 tokens, f16 overflow would then require max|V| > 3060, which is not possible - and all runs still produced garbage. So f16 overflow is not (the only) mechanism. Side observation: the same offset change flipped the 76,502-token point from CLEAN to GARBAGE, suggesting the band boundaries sit on a numerically delicate margin (consistent with the boundary also differing between the two builds).
Related
- Eval bug: GLM-4.7-Flash flash attention error on (long?) prompts with KV quantization #19036 - non-contiguous V tensor broke FA at >=65536 tokens (fixed)
- Bug: Flash Attention performs worse under ROCM #10439 - flash attention performance under ROCm
- Discussion RDNA4 Llama Experiments — Squeezing Every Token/s from the R9700 #21043 - R9700 performance experiments
Activity
DanoPTT commentedon Oct 20, 2026
Same card (R9700, ROCm 10.0, Windows) and Qwen3.8-27B, with your generator unchanged, greedy,
cache_prompt: false. With raw/completionI also get the immediate EOS, but not in a band: it shows up at 10k, 30k and everything from 40k to 94k, while 4k and 20k answer correctly. The same prompt sent as a user message through/v1/chat/completionsanswers "AMBERFROST9317" at every length I tried, including 66,338 and 87,484 tokens (the same padded KV lengths as yours, 66,560 and 87,552), with the first token at logprob −0.001.So I don't think the logits are corrupted. In the raw runs the top token is
<|im_end|>, followed by<|endoftext|>or\n\n. Without a chat template the instruct model simply ends its turn after "Reply with only the codename". At 30k the margin is small (−0.31 vs −1.34), so any small numerical change can flip it, which would also explain theFATTN_KQ_MAX_OFFSETresult and the difference between your two builds. Could you try it through the chat endpoint? If it still fails there, it's a different problem and worth digging into.One caveat: I'm not on plain upstream but on a fork with RDNA patches (stew675 rdna-boosts r92), which has different FA kernels. The template also adds ~12 tokens, so the last ubatch has a different shape.
Build: llama.cpp fork (stew675 rdna-boosts r92 on 097cf33), R9700 / gfx1201, ROCm 10.0, Windows. Qwen3.8-27B UD-Q5_K_XL (V2 requant),
-fa on, KV q8_0,-ub 512 -b 1024, ctx 262144.| N | prompt tokens | raw
/completion| first-token top-3 (raw; "" =<|im_end|>/<|endoftext|>) |/v1/chat/completions||---|---|---|---|---|
| 40 | 4,003 | OK | — | OK |
| 100 | 9,991 | EOS | "" −0.011,
\n\n−5.21, "" −6.31 | || 200 | 19,963 | OK (
\n\n<think>… correct) |\n\n−0.076, "" −2.68," Reply"−5.89 | || 300 | 29,935 | EOS | "" −0.312,
\n\n−1.341," Reply"−5.89 | OK || 400 | 39,907 | EOS | "" −0.031,
\n\n−3.77, "" −5.10 | || 500 | 49,879 | EOS | "" −0.029,
\n\n−3.69, "" −5.98 | || 600 | 59,851 | EOS | "" −0.005, "" −5.90,
\n\n−6.22 | || 665 | 66,326 | EOS | "" −0.011, "" −4.68,
\n\n−7.23 | OK (66,338 tok, "AM" −0.001) || 720 | 71,814 | EOS | "" −0.005, "" −5.97,
\n\n−6.50 | || 767 | 76,502 | EOS | "" −0.005,
\n\n−5.89, "" −6.67 | || 820 | 81,786 | EOS | "" −0.008, "" −5.11,
\n\n−6.45 | || 877 | 87,472 | EOS | "" −0.006, "" −5.55,
\n\n−6.61 | OK (87,484 tok, "AM" −0.001) || 940 | 93,751 | EOS | "" −0.006, "" −5.60,
\n\n−6.39 | |Chat request:
{"messages":[{"role":"user","content":<prompt>}], "temperature":0, "max_tokens":300, "chat_template_kwargs":{"enable_thinking":false}}. In the raw runs the top token was<|im_end|>(id 248046), the other empty one<|endoftext|>(id 248044), checked withreturn_tokens+/detokenizeat N=100 and N=665.konopka commentedon Oct 20, 2026
@DanoPTT chat endpoint /v1/chat/completions seems to be working.
Results:
"AMBERFROST9317", finish_reason=stop, 10 completion tokens.
tw-cook commentedon Oct 22, 2026
Comment draft for llama.cpp issue #30884 (2026-10-21)
Additional repro on gfx1201 (Linux, ROCm 10.1) — garbage bands appear at ~1/10 the reported context lengths
Reproducible with Qwen3.8-27B (head_dim 256, GQA 6, 16 full-attention layers of 65, hybrid) on 2× R9700 (gfx1201), Linux, ROCm 10.1, llama.cpp built from
41abbfd59(2026-10-10) andb76b13379(2026-10-15) — both before b11755. Band positions differ from the OP's, but the signature is identical (overconfident first token, immediate stop, correct answer not in the top 10).Server:
Using the OP's exact prompt generator, temp 0, n_predict 300 (note: this model is a thinking model — the reasoning preamble consumes ~75 tokens, so short n_predict reads as "off-task" and is a false positive):
Deterministic: same binary + same prompt → same result. The garbage first token is the model's empty-byte EOS token at the top of the distribution (lp ≈ -0.04..-0.66) with "\n\n" at ≈ -1..-2 and the answer nowhere in the top 10 — the same shape as the OP's 87,472-token run. Within a padded 256-window the failure mode varies with exact length: at 5,999 it's immediate EOS; at 6,121 (same 24×256 window) the model echoes the question verbatim for ~20–60 tokens, then EOS.
A/B on the same hardware:
b76b13379(contains the FA tile-race fix CUDA/HIP: fix race in flash_attn_ext_f16_process_tile when nbatch_combine != DKQ/2 #30103 and the cuda : fix -Wtautological-compare in fattn vec dispatch #30519/ggml-cuda: fattn do not check oob if k_VKQ_0 = (k_VKQ_max - nbatch_fa) #30572/CUDA: fix shared-memory WAR hazard in flash_attn_ext_vec #30573 FA fixes) shows the identical Qwen band → none of those fixes resolve it.Our garbage first token is produced by the final ~1,900-token ubatch (ub=2048) — not the OP's 4-token TILE tail (ub=288) — so both nq shapes appear affected on Qwen.
Happy to share the exact test script (prompt generator + sweep harness) if it would help with the investigation.