Repository navigation
qsa: the default block-scores kernel without #2234's per-query keys (a gfx1151 verify window stalled on v0.1.45); the batch rows get their own kernel - #3151
Closed
gt50 wants to merge 1 commit into
Conversation
…y keys; the batch rows get their own kernel f629fb0 (Niko1221#2234) gave block_scores_multi_kernel, the kernel every decode window's QSA block scores run on, a by-value QsaScoreRows argument (16 pointers and a count) and a per-query branch for the opt-in STRATA_BATCH_ATTN_ROWS path. With them, a Radeon 8060S (gfx1151, Windows, ROCm 10.2.0a20260930) stalls in its first verify window ("an earlier window never finished on the GPU (Niko1221#267)", the GPU rang 0) when gfx1151's default switches are on: it needs STRATA_PLE_BATCH, STRATA_LFUSE and STRATA_TSUM or STRATA_GDN_SPLIT beside it. It does not stall every time: 6 of 7 starts of one build stalled (c8265fc, the commit just before v0.1.45 on main), and every start of v0.1.45 with all the defaults on. block_scores_multi_body is a template on OWN: the default kernel (OWN false) is again the pre-Niko1221#2234 kernel, with no QsaScoreRows argument and no branch, and qsa_block_scores_rows launches block_scores_multi_rows_kernel (OWN true). Checked on that Radeon 8060S, UD-IQ4_XS, the 19 gfx1151 defaults on: - bisected to f629fb0's qsa_select.cu change alone (3 starts each step); - with this change: 5 of 5 starts answer; - batch_rows_parity (qsa_block_scores_rows at 1/2/4/8 rows against one call per row, bitwise), qsa_parity, qsa_topk_parity, hip_native_qsa_score and verify_batch_parity pass; - three greedy answers (284-344 tokens) byte-identical to v0.1.45 with STRATA_PLE_BATCH=0, decode 34.9 / 48.9 / 40.3 tok/s against 34.9 / 48.5 / 40.8. Not built for CUDA or SYCL here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On v0.1.45 a Radeon 8060S (gfx1151, Windows 11, ROCm 10.2.0a20260930) stalls in its first verify window when gfx1151's default switches are on. The engine prints
verify: an earlier window never finished on the GPU (#267)and "the GPU rang 0", the watchdog restarts it after 60 s, and no token comes out. It happens with a stock config and with mine. The installed 0.1.40.2 engine, with the same switches and config, does not stall.Cause: f629fb0 (#2234), its
qsa_select.cuchange.block_scores_multi_kernelis the kernel every decode window's QSA block scores run on. #2234 gave it a by-valueQsaScoreRowsargument (16 pointers and a count) and a per-query branch for the opt-inSTRATA_BATCH_ATTN_ROWSpath. Both apply with that switch off.Fix:
block_scores_multi_bodyis a template onOWN.OWNfalse) is the pre-Batch decode: one launch for all rows in QSA attention and GDN recurrence (opt-in) #2234 kernel again, with noQsaScoreRowsargument and no branch.qsa_block_scores_rowslaunches a newblock_scores_multi_rows_kernel(OWNtrue).src/kernels/cuda/qsa_select.cu, +33/−14.How it was found
All on that Radeon 8060S, Qwen3.8-Flash-Next UD-IQ4_XS,
--spec 4 --mtp,--kv int8. Each trial was a fresh start, then a 14-token request withmax_tokens8. No answer in 30 s counted as a stall; a normal answer takes 0.4-0.7 s.STRATA_GFX1151_DEFAULTS=0and subsets turned on.STRATA_PLE_BATCHis on in every set that stalled. It needsSTRATA_LFUSEandSTRATA_TSUMorSTRATA_GDN_SPLITbeside it, and other switches change which set is enough. With the other 18 defaults on,STRATA_PLE_BATCH=0stops the stall.qsa_decode_attn.cu; f755a06 fixes that). So each part was applied alone to the merge base 296b59d:qsa_decode_attnfiles (f755a06's refactor)verify_kernels)qsa_select.cu/.hppchangeThe f-145-fix3 merge with #2622 answered 3 of 3, so the PLE gather change is not the cause. Several 0.1.45 merges do not build on Windows HIP without e6ad6c8's 4-line
hip_compatfix, which these tests applied.Checked with this change
On v0.1.45 (477ac13) plus this commit, on the same machine:
batch_rows_paritypasses (it checksqsa_block_scores_rowsat 1, 2, 4 and 8 rows against one call per row, bitwise). So doqsa_parity,qsa_topk_parity,hip_native_qsa_scoreandverify_batch_parity.qsa_scores_gfx1100_paritybuilds and skips itself, since it is gfx1100 only.STRATA_PLE_BATCH=0.Not covered
if constexpr, as other.cufiles here already do.--batchandSTRATA_BATCH_ATTN_ROWS=1was not measured. Only its parity test ran.tools\hip\build_windows.bat testsstops athc_persist_parity.cpp. It uses POSIXsetenv/unsetenv, which this change does not touch.Found while testing #1916 on v0.1.45. cc @yi-sudo (#2234)
🤖 Generated with Claude Code