Repository navigation
Conversation
…ot fold (it expanded each dp4a into ~24 ops) The wide (multi-column) Q4_K / Q5_K kernels take the activation sum of the min term as dp4a(0x01010101, u). With the literal constant IGC (oneAPI 2026.1, DG2) does not emit a dp4a: it expands each one into byte movs + add3 (16 movs + 8 add3 per SIMD16 half), ~60% of the 4-column kernel's integer instructions (offline ISA, ocloc acm-g10: 652 -> 320 integer instructions per loop iteration). For 2+ columns the kernels now compute one4 = 0x01010101 + (n_in >> 31), which is 0x01010101 for every n_in >= 0 but a value the compiler cannot prove, and pass it down (WideIQ4XS takes the same argument). One column keeps the literal: there it measured 4-14% faster. The same integers, so the same bits. A770 dense_bench on main, the same output hash in every row: - Q4_K: 1 column -0.2..+5%; 2 columns +6..+12%; 3 columns +10..+16%; 4 columns +12..+18% (e.g. QSA q 145.9 -> 171.7 GB/s, out-proj 139.4 -> 164.9). - Q5_K: 1 column the same (the small k/v and out-proj rows swing +-15% run to run on both builds); 2 columns -2..+3%, k/v -11%; 3 columns +10..+19%; 4 columns +3..+13% (QSA q 162.3 -> 183.1). The engine's verify window runs mostly 4 columns. Engine on a fork tree (with Niko1221#2207; Coder IQ1_M, 8K code prompt, 3 pairs): GPU wait 32.07 / 31.93 / 31.94 -> 31.85 / 31.63 / 31.59 ms a round, reference tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Tested on two Arc Pro B70s (xe) with a layer split (26) and
That is +0.8% / +1.0% / +1.0%, smaller than the A770 kernel numbers but the same in both runs (the runs without it differ by 0.1 tok/s). On five greedy prompts the gain is +0.8% to +2.1%, with text identical to the build without it. Prompt reading is unchanged. Together with #2752 and #2746: 89.8 / 131.4 / 100.0, pipelined text the same as serial, and two runs of a 148K prompt followed by 20 prompts of 13K with no lost device and no xe reset. We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine. |
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
sycl: Q4_K/Q5_K wide GEMV - pass dp4a's ones word as a value IGC cannot fold (it expanded each dp4a into ~24 ops)
The wide (multi-column) Q4_K / Q5_K kernels take the activation sum of the min term as dp4a(0x01010101, u). With the
literal constant IGC (oneAPI 2026.1, DG2) does not emit a dp4a: it expands each one into byte movs + add3 (16 movs +
8 add3 per SIMD16 half), ~60% of the 4-column kernel's integer instructions (offline ISA, ocloc acm-g10: 652 -> 320
integer instructions per loop iteration). For 2+ columns the kernels now compute one4 = 0x01010101 + (n_in >> 31),
which is 0x01010101 for every n_in >= 0 but a value the compiler cannot prove, and pass it down (WideIQ4XS takes the
same argument). One column keeps the literal: there it measured 4-14% faster. The same integers, so the same bits.
A770 dense_bench on main, the same output hash in every row:
GB/s, out-proj 139.4 -> 164.9).
k/v -11%; 3 columns +10..+19%; 4 columns +3..+13% (QSA q 162.3 -> 183.1).
The engine's verify window runs mostly 4 columns. Engine on a fork tree (with sycl/A770: wide MMVQ kernels at 2-8 columns in large-GRF mode on Arc A-series (bit-identical; 1.3-7.4x kernel, decode +10-17%) #2207; Coder IQ1_M, 8K code prompt,
3 pairs): GPU wait 32.07 / 31.93 / 31.94 -> 31.85 / 31.63 / 31.59 ms a round, reference tokens.
🤖 Generated with Claude Code