Skip to content

[SYCL] support MUL_MAT and OUT_PROD with Q1_0 - #24721

Merged
ggerganov merged 1 commit into
ggml-org:masterfrom
arthw:support_q1_0
Jun 18, 2026
Merged

ggerganov merged 1 commit into
ggml-org:masterfrom
arthw:support_q1_0

Conversation

@arthw

@arthw arthw commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor

Implement the feature request: #21641

support MUL_MAT and OUT_PROD with Q1_0.
all related UT cases are passed.

Test LLM: Bonsai-1.7B-Q1_0.gguf

on B60

Test Flash Attn pub build (t/s) PR build (t/s) Speedup
pp512 fa = 0 155.23 ± 0.40 3041.99 ± 4.22 19.6x
pp512 fa = 1 153.45 ± 0.06 2685.07 ± 1.75 17.5x
tg128 fa = 0 60.16 ± 0.12 148.47 ± 0.13 2.47x
tg128 fa = 1 65.74 ± 0.07 159.67 ± 0.54 2.43x

on Xe GPU in Ultra 7 265K

Test Flash Attn pub build (t/s) PR build (t/s) Speedup (PR vs pub)
pp512 fa = 0 131.42 ± 0.34 472.65 ± 0.06 3.60x
pp512 fa = 1 114.29 ± 0.14 291.50 ± 0.05 2.55x
tg128 fa = 0 13.12 ± 0.00 16.67 ± 0.00 1.27x
tg128 fa = 1 16.25 ± 0.01 21.83 ± 0.00 1.34x

On UHD 770 in i7-13700K

Test fa Base t/s Primary t/s Increase Rate (Primary vs Base)
pp512 0 196.56 212.32 8.02%
pp512 1 122.01 129.45 6.10%
tg128 0 24.42 18.39 -24.69%
tg128 1 27.26 19.23 -29.46%

In base, 400% CPU usage, PR use 100% CPU.
the result of base comes from CPU, instead of GPU.
So UHD 770 replace 3 CPU cores in above result.

@arthw
arthw requested a review from a team as a code owner June 17, 2026 13:38
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Jun 17, 2026
@arthw arthw added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Jun 18, 2026
@ggerganov
ggerganov merged commit dd69db2 into ggml-org:master Jun 18, 2026
51 of 54 checks passed
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants