You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Misc. bug: Vulkan performance regression on Iris Xe #12754
Which llama.cpp modules do you know to be affected?
llama-cli
Command line
llama-cli -m "modelname.gguf" -p "prompt" -ngl 50
Problem description & steps to reproduce
Under Windows 11, taking Mistral-Nemo-Instruct_2407 Q4_K_M as reference, performance went down from 4.7 tok/s with b5017 to 4.3 tok/s b5050 (same intel drivers 32.0.1016651)
For gemma-3-4b-it-Q6-K performance is the same in both builds at 9.7 tok/sec
While this may be anecdotal given Iris Xe is a rather basic integrated CPU, b5017 was the first version where I noticed llama.cpp Vulkan ran faster than AVX-2 on an i7 1165G7 (this did not use to be the case, Vulkan was quite more sluggish before), and running inference on the ris Xe is quite energy efficient : laptop can run inference at 100% GPU with fans off.
Reporting this in case the Vulkan wizards can take advantage of that data point!
With all the recent enhancements to vulkan in llama.cpp, it's now rather comfortable to run small-ish models on regular laptops.
The improvement came from #12135, but sadly I missed a bug that made that build both fast and incoherent. The fix (#12722) to the incoherence reduced the performance gain. I hope to get a good chunk of the performance back with further tuning and have had some promising first results, but I'm not yet done with that.
Activity
0cc4m commentedon Apr 4, 2025
The improvement came from #12135, but sadly I missed a bug that made that build both fast and incoherent. The fix (#12722) to the incoherence reduced the performance gain. I hope to get a good chunk of the performance back with further tuning and have had some promising first results, but I'm not yet done with that.
EricGrange commentedon Apr 4, 2025
Thanks for your efforts! They are appreciated, they make LLMs literally cool & quiet ;)
0cc4m commentedon Apr 5, 2025
#12767 should bring some of the performance gains back for you, I hope.
EricGrange commentedon Apr 5, 2025
Yes, performance gains are back on Iris Xe here for Mistral Nemo, and perf on Gemma 3 unchanged.
0cc4m commentedon Apr 5, 2025
Yes, this only affects the legacy quants, not iq or k-quants.