[rafatxf]

On Turing the Q2_0 matvec of the decode windows is far from the card's bandwidth: the QSA layers' attn_q (2560 → 12288, Q2_0 in the canonical pack) takes 80 µs at three columns, ~110 GB/s on an RTX 2080 Ti. The exact layout gives each row four warps and half a 64-value block per thread, then a cross-warp reduction; at the engine's 2,560 inputs a row has 40 blocks, so 48 of the 128 threads have no chunk and the reduction is most of the kernel.

q2_0_warp_rows_kernel gives a warp four rows. Lane l runs the chunks of the exact kernel's threads l, 32 + l, 64 + l and 96 + l, in their loop order, adds the four sums the way warp 0 adds the other warps' partials, then the same warp_sum. Every output is bit for bit native_q2_0_mmvq_kernel's (one column) and launch_multi's exact layout (two to four columns); the upstream (non-exact) layout keeps its kernel.

Default on sm_75 only (the only card I can measure); STRATA_Q2_WARP=0 keeps the old kernels, =1 takes this one on any CUDA card.

Setup

RTX 2080 Ti 22 GB (sm_75), i7-6700, CUDA 13.0, SM 1650 MHz, 240 W. Flash-Next, canonical Q2_0 pack, --max-context 524288, --serve.

Kernel (2560 × 12288, cold weights: six copies in rotation, 300 calls)

columns	exact layout	warp rows
1	36.6 µs	24.8 µs	1.48×
2	59.8 µs	31.4 µs	1.90×
3	68.3 µs	31.3 µs	2.18×
4	88.4 µs	36.6 µs	2.42×

The outputs are compared with memcmp against the engine's own launch for every column count: identical. One and two rows per warp, and eight warps per block, were measured too; four rows and four warps was best.

Engine

Together with the QSA attention change of #2220 (both bitwise, measured as one step), 512 new tokens, two runs each: Spanish chat 86.0 → 87.8 t/s, code 103.1 → 106.1, English at 32K 75.3 → 78.4. The same kernel also serves the shared expert's Q2_0 gate/up on its side stream.

Part of the Turing set tracked in #2197.

🤖 Generated with Claude Code

[Niko1221]

Thanks rafatxf, good work. We ported this with your authorship kept. We have no Turing card, so we could not confirm the speed. What we did check: the default path is byte-identical to before (identity gate on RTX 3060, P100 and A4000), and with the new path forced on, including a build that runs the real sm_75 device code on an sm_86 card, the gate is identical too. Because we cannot measure it on sm_75 it ships as opt-in (STRATA_SM75_PERF=1, plus your own variable STRATA_Q2_WARP), documented in docs/DETAILS.md. If a Turing owner confirms the gain we will consider turning it on by default.

[rafatxf]

Ported in 0.1.45, thanks! Turing numbers for STRATA_SM75_PERF (2x RTX 2080 Ti, Q2_0 and IQ3_S) are in #2069. Closing.