[rafatxf]

The decode attention over the QSA selection (attn_chunk_kernel<1>, INT8 pools) saturates the SM's shared-memory pipe on Turing. Its time grew linearly with the queries while the SMs were half empty: per cell, every warp reads two 16-byte slices of each of the 12 query heads from shared memory (a two-way bank conflict at lane * 8) and runs 12 warp_sums, 60 shuffles; the value loop then reads the 12 probabilities of a cell as 12 scalar loads. Loading the K/V codes earlier (software prefetch) changed nothing, so it is not memory latency.

attn_chunk_q8_v3_kernel does the same operations in the same order:

• the lane's slice of the 12 heads (8 dimensions each) is read from shared memory once, into registers;
• the 12 lane sums are reduced with reduce12_gfx906 (16 shuffles; every sum is warp_sum's pairs, as its comment says);
• the probabilities are stored cell-major, so the value loop reads a cell's 12 as three 16-byte broadcasts.

The chunk partials and the merged output are bit for bit the default kernel's. It is used by both entry points (qsa_decode_attn_batch, qsa_decode_attn_step), so the MTP drafter's attention gets it too. Default on sm_75 only; STRATA_ATTN_V3=0 keeps the default kernel, =1 takes this one on any CUDA card. Lane-cell scoring (STRATA_ATTN_LANECELL=1) keeps its path.

Setup

RTX 2080 Ti 22 GB (sm_75), i7-6700, CUDA 13.0, SM 1650 MHz, 240 W. Flash-Next, canonical Q2_0 pack, --kv int8, --max-context 524288, --serve.

Kernel (chunk kernel + merge, synthetic INT8 pools, permuted page table, 200 calls)

selected cells	queries	default	v3
2,051	1	51.6 µs	34.5 µs	1.50×
2,051	2	78.5 µs	41.0 µs	1.91×
2,051	3	110.0 µs	73.0 µs	1.51×
2,051	4	145.6 µs	84.9 µs	1.71×
600	1	45.9 µs	28.6 µs	1.60×
600	2	74.6 µs	32.8 µs	2.27×
600	3	105.5 µs	33.5 µs	3.15×
600	4	136.9 µs	33.3 µs	4.11×

Partials and outputs compared with memcmp against the default kernel in every case: identical. With the swizzle and reduce12 alone (the gfx906 variants, also bitwise on CUDA) the 2,051-cell / 3-query case is 78 µs; the register copy of the query is what takes it to 73 µs, and further at fewer cells. At 2,051 cells and 3-4 queries the registers allow two blocks per SM (198-264 blocks: two waves); forcing three spills.

qsa_prompt_attn_parity reports the same errors with STRATA_ATTN_V3=0 and 1 (its reference is this decode kernel: int8 32K / 2,048 queries 71.5 → 36.0 ms per chunk).

Engine

Together with the Q2_0 matvec change of #2221 (both bitwise, measured as one step), 512 new tokens, two runs each: Spanish chat 86.0 → 87.8 t/s, code 103.1 → 106.1, English at 32K 75.3 → 78.4.

Part of the Turing set tracked in #2197.

🤖 Generated with Claude Code

[Adamyno]

Same-day verification of this PR (merged onto v0.1.43.1 = a0116b27, together with #2221, #2222, #2234) on hardware the PR author doesn't have: 3× RTX 2080 Ti 22 GiB (sm_75), layer split 15,32, i9-7980XE, 125 GiB RAM, Linux, CUDA 13.0, driver 580.178.04, Flash-Next Q4_K_XL, 524,288-token context, --kv int8 --spec 3 --spec-min-p 0.54 --batch 2.

Correctness gates: ctest 7/7 passed; qsa_parity, s_gemv_parity, native_multi_parity, iq_parity (fresh fixtures from tools/iq_fixture.py) — 0 failures.

A/B, same prompts, temperature 0, both binaries built from source the same hour (tools/strata_bench.py, engine-side timings, medians):

Metric	baseline v0.1.43.1	+ this PR stack	delta
chat decode tok/s (n=9)	60.1	62.4	+3.8 %
agent decode tok/s (n=15)	66.8	70.1	+4.9 %
4K prompt tok/s	586.5	586.3	-0.0 %
16K prompt tok/s	1244.4	1243.4	-0.1 %
TTFT (chat/4K/16K/agent)	—	unchanged	within ±0.3 %

Prefill and TTFT sitting exactly on the noise floor matches the PRs' scope (decode-window kernels only) and says the A/B is clean. Decode medians move in the predicted direction, but the single-run spread on this rig (one noisy chat run swung -13 %) keeps this an indicator, not proof.

Not tested: --batch 4/8 (needs more slots than this config's 2 — a separate batch run measured #2234 alone at +3.4 % burst), --batch-mtp, 1-GPU / 2-GPU splits, Windows.

[Niko1221]

Thanks rafatxf, good work. We ported this with your authorship kept. We have no Turing card, so we could not confirm the speed. What we did check: the default path is byte-identical to before (identity gate on RTX 3060, P100 and A4000), and with the new path forced on, including a build that runs the real sm_75 device code on an sm_86 card, the gate is identical too. Because we cannot measure it on sm_75 it ships as opt-in (STRATA_SM75_PERF=1, plus your own variable STRATA_ATTN_V3), documented in docs/DETAILS.md. If a Turing owner confirms the gain we will consider turning it on by default.

[rafatxf]

Ported in 0.1.45, thanks! Turing numbers for STRATA_SM75_PERF (2x RTX 2080 Ti, Q2_0 and IQ3_S) are in #2069. Closing.