Repository navigation
Feature Request: SYCL backend support for newly added Q1 quantization #21641
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
SYCL backend support for newly added Q1 quantization. This would allow higher token/sec for normal igpu variants. the Bonsai 1.7 B model gives like 1 tok/sec for this with normal cpu, adding igpu support can make this 10 tok/sec i believe. which would be great. @khosravipasha
Motivation
Q1 quantization has been introduced and adopted in Bonsai models (by Prism ML), offering extremely aggressive compression with minimal memory footprint. The performance of these models is pretty great and i believe I can run this for daily tasks on my desktop which doesn't have dedicated gpu.
Possible Implementation
No response
Reacted by Jairo Llopis and UmbraAtrox
Activity
NeoZhangJianyu commentedon Apr 9, 2026
@vanshnawander
What's the target Intel GPU for your requirement?
Could you share the detail model of the iGPU or CPU?
Is it the performance target in your case: get 10 t/s on 1.7 B model?
Thank you!
vanshnawander commentedon Apr 9, 2026
clinfo)GGML_SYCL=ONwith Intel oneAPIDetected Devices (from
sycl-ls)[opencl:gpu:0]I want something faster, becuase it's currently only cpu based it is giving 1 tok/sec, i believe sycl would give a significant speed up boost.
Thank you!
khosravipasha commentedon Apr 9, 2026
The CPU optimization for x86 is not merged yet, right now only fast on Metal and Mac NEON path.
After this PR is merged probably will get much better CPU speeds.
#21636
Not too familiar with SYCL.
There is some works on Vulkan backend from the community as well in our fork after its stable I can ask them to send a PR here. PrismML-Eng#9
And had some partial success with OpenCL (this is old fork so have not tested it recently):
https://github.com/PrismML-Eng/llama.cpp/tree/prism-android
vanshnawander commentedon Apr 9, 2026
Thanks for the update, I will check that out. I would also love to contribute to this, I will check if i can help in the Vulkan/OpenCL backend part.
NeoZhangJianyu commentedon Apr 10, 2026
It's clear to us.
It's interesting to support Q1 type.
I can't estimate the workload to support this Q1 model.
If there is reference like CUDA implementation, it will help estimate the workload.
Thank you!
khosravipasha commentedon Apr 10, 2026
We have a PR for cuda here
#21629
What kind of hardware SYCL supports? Happy to help with testing if I have the hardware.
Just saw a vulkan backend was merged as well recently.
NeoZhangJianyu commentedon Apr 11, 2026
SYCL backend support all Intel dGPU and iGPU since 10th Core CPU.
We will discuss this feature later.
Thank you!
github-actions commentedon May 26, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
arthw commentedon Jun 17, 2026
@vanshnawander @khosravipasha
We are planning to handle this feature.
arthw commentedon Jun 17, 2026
@khosravipasha @vanshnawander
It's implemented by PR: #21641
khosravipasha commentedon Jun 17, 2026
Thanks, This is awesome :D
vanshnawander commentedon Jun 18, 2026
Thank you so much!!
github-actions commentedon Aug 2, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.