Skip to content

sycl: Q4_K/Q5_K wide GEMV - pass dp4a's ones word as a value IGC cannot fold (it expanded each dp4a into ~24 ops) - #2753

Closed
aslater3 wants to merge 1 commit into
Niko1221:mainfrom
aslater3:a770/up-mmvq-one4
Closed

aslater3 wants to merge 1 commit into
Niko1221:mainfrom
aslater3:a770/up-mmvq-one4

Conversation

@aslater3

Copy link
Copy Markdown
Contributor

sycl: Q4_K/Q5_K wide GEMV - pass dp4a's ones word as a value IGC cannot fold (it expanded each dp4a into ~24 ops)

The wide (multi-column) Q4_K / Q5_K kernels take the activation sum of the min term as dp4a(0x01010101, u). With the
literal constant IGC (oneAPI 2026.1, DG2) does not emit a dp4a: it expands each one into byte movs + add3 (16 movs +
8 add3 per SIMD16 half), ~60% of the 4-column kernel's integer instructions (offline ISA, ocloc acm-g10: 652 -> 320
integer instructions per loop iteration). For 2+ columns the kernels now compute one4 = 0x01010101 + (n_in >> 31),
which is 0x01010101 for every n_in >= 0 but a value the compiler cannot prove, and pass it down (WideIQ4XS takes the
same argument). One column keeps the literal: there it measured 4-14% faster. The same integers, so the same bits.

A770 dense_bench on main, the same output hash in every row:

  • Q4_K: 1 column -0.2..+5%; 2 columns +6..+12%; 3 columns +10..+16%; 4 columns +12..+18% (e.g. QSA q 145.9 -> 171.7
    GB/s, out-proj 139.4 -> 164.9).
  • Q5_K: 1 column the same (the small k/v and out-proj rows swing +-15% run to run on both builds); 2 columns -2..+3%,
    k/v -11%; 3 columns +10..+19%; 4 columns +3..+13% (QSA q 162.3 -> 183.1).
    The engine's verify window runs mostly 4 columns. Engine on a fork tree (with sycl/A770: wide MMVQ kernels at 2-8 columns in large-GRF mode on Arc A-series (bit-identical; 1.3-7.4x kernel, decode +10-17%) #2207; Coder IQ1_M, 8K code prompt,
    3 pairs): GPU wait 32.07 / 31.93 / 31.94 -> 31.85 / 31.63 / 31.59 ms a round, reference tokens.

🤖 Generated with Claude Code

…ot fold (it expanded each dp4a into ~24 ops)

The wide (multi-column) Q4_K / Q5_K kernels take the activation sum of the min term as dp4a(0x01010101, u). With the
literal constant IGC (oneAPI 2026.1, DG2) does not emit a dp4a: it expands each one into byte movs + add3 (16 movs +
8 add3 per SIMD16 half), ~60% of the 4-column kernel's integer instructions (offline ISA, ocloc acm-g10: 652 -> 320
integer instructions per loop iteration). For 2+ columns the kernels now compute one4 = 0x01010101 + (n_in >> 31),
which is 0x01010101 for every n_in >= 0 but a value the compiler cannot prove, and pass it down (WideIQ4XS takes the
same argument). One column keeps the literal: there it measured 4-14% faster. The same integers, so the same bits.

A770 dense_bench on main, the same output hash in every row:
- Q4_K: 1 column -0.2..+5%; 2 columns +6..+12%; 3 columns +10..+16%; 4 columns +12..+18% (e.g. QSA q 145.9 -> 171.7
  GB/s, out-proj 139.4 -> 164.9).
- Q5_K: 1 column the same (the small k/v and out-proj rows swing +-15% run to run on both builds); 2 columns -2..+3%,
  k/v -11%; 3 columns +10..+19%; 4 columns +3..+13% (QSA q 162.3 -> 183.1).
The engine's verify window runs mostly 4 columns. Engine on a fork tree (with Niko1221#2207; Coder IQ1_M, 8K code prompt,
3 pairs): GPU wait 32.07 / 31.93 / 31.94 -> 31.85 / 31.63 / 31.59 ms a round, reference tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@deriknel

Copy link
Copy Markdown

Tested on two Arc Pro B70s (xe) with a layer split (26) and --pipeline-windows 2, Swift 1.5 Flash-Next IQ3_S with every expert resident and MTP drafting (3 draft tokens, so the verify window reads 4 columns), engine 0.1.44.1 (296b59d) plus our open SYCL PRs (#2433, #2436, #2437) and one local kernel change. This model has 47 Q4_K and 35 Q5_K tensors outside the experts. Each arm on a fresh engine, the arms alternated, two runs each, decode tok/s:

prose code long
without 83.7 / 83.6 122.8 / 122.7 93.3 / 93.3
with this PR 84.3 / 84.3 124.0 / 124.0 94.3 / 94.2

That is +0.8% / +1.0% / +1.0%, smaller than the A770 kernel numbers but the same in both runs (the runs without it differ by 0.1 tok/s). On five greedy prompts the gain is +0.8% to +2.1%, with text identical to the build without it. Prompt reading is unchanged. Together with #2752 and #2746: 89.8 / 131.4 / 100.0, pipelined text the same as serial, and two runs of a 148K prompt followed by 20 prompts of 13K with no lost device and no xe reset.

We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.

@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

  1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
  2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
  3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
  4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

@Niko1221 Niko1221 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants