[chaxu01]

Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27
Change-Id: I406a6606c3fea51f8cd32331c1ed71466845d363

Overview

This PR adds KleidiAI SME2 matmul support for Q4_K and enables Q6_K by mapping it to the existing Q8_0 path. It also includes related cleanup and fixes a latent F32 matmul bug in multi-sequence handling that was exposed by the newer Kimi-K3 multi-sequence test.

Additional information

Llama-3.2-1B — PP512 Benchmarks for M4Pro (t/s)

Threads	Q4_K_M w/ KleidiAI	Q4_K_M w/o KleidiAI	Q6_K w/ KleidiAI	Q6_K w/o KleidiAI
1	387.37	109.46	763.70	92.18
2	772.42	220.31	1567.80	182.40
4	771.46	435.41	2083.68	360.34
6	725.49	584.96	1991.06	483.47
8	863.82	699.78	2402.58	567.84

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, Codex was used for testing and verification.

[chaxu01]

@ggerganov Could you please take a look when you have a chance? Thanks!

[chaxu01]

@ggml-org/maintainers Could someone please take a look at this PR when you have a chance?

This adds KleidiAI SME2 support for Q4_K and Q6_K, with significant performance improvements demonstrated on M4 Pro.

Any feedback would be appreciated. Thanks!

[taronaeo]

@chaxu01 Do you have merge rights? :)

[max-krasnyansky]

Still seems to be doing a bit worse than the MM_INT8 kernels, especially on longer prompts for me.
(still as in from my original tests way back)

M4 Pro, master branch as if right now, Qwen3.8-4B (distilled) in Q4_K_M

First the default build, cmake -B build-macos -G Ninja

./build-macos/bin/llama-completion --jinja -st -t 10 -m ../gguf/Qwen3.8-4B-Q4_K_M.gguf -f ../context_3000_test.txt -n 128 --device none
...
0.01.650.743 I system_info: n_threads = 10 (n_threads_batch = 10) / 14 | MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | SME = 1 | ACCELERATE = 1 | REPACK = 1 | 
...
0.22.120.142 I common_perf_print: prompt eval time =   17949.97 ms /  4004 tokens (    4.48 ms per token,   223.06 tokens per second)
0.22.120.142 I common_perf_print:        eval time =    2538.86 ms /   127 runs   (   19.99 ms per token,    50.02 tokens per second)

This PR locally rebased with the same master as above,
cmake -B build-macos-kai -G Ninja -D GGML_CPU_KLEIDIAI=ON

GGML_KLEIDIAI_SME=3 ./build-macos-kai/bin/llama-completion --jinja -st -t 10 -m ../gguf/Qwen3.8-4B-Q4_K_M.gguf -f ../context_3000_test.txt -n 128 --device none
...
0.03.620.373 I system_info: n_threads = 10 (n_threads_batch = 10) / 14 | MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | SME = 1 | ACCELERATE = 1 | KLEIDIAI = 1 | REPACK = 1 | 
...
0.29.085.232 I common_perf_print: prompt eval time =   21799.13 ms /  4004 tokens (    5.44 ms per token,   183.68 tokens per second)
0.29.085.233 I common_perf_print:        eval time =    3643.14 ms /   127 runs   (   28.69 ms per token,    34.86 tokens per second)

Tried GGML_KLEIDIAI_SME=1,2,3 and auto
3 seems to be the best. 10 compute threads seems to be getting mapped to the 10 perf cores.

This is just FYI. I no objections to merging. The outputs from the models look good.

[chaxu01]

@chaxu01 Do you have merge rights? :)

I used to have merge rights, but it looks like I no longer do. :)

[ggerganov]

@chaxu01 Do you have merge rights? :)

I used to have merge rights, but it looks like I no longer do. :)

@chaxu01 I just moved you to the @ggml-org/ggml-cpu maintainer team - should have rights now.

[chaxu01]

@max-krasnyansky Thanks Max, appreciate the additional testing and feedback.

Interesting to see the native MM_INT8 path performing better for this workload. My original benchmarks with Llama-3.2-1B Q4_K_M and PP512 showed significant improvements with KleidiAI, so the performance difference may be related to the model size, prompt length, and thread configuration.

It would be worth investigating this further, particularly the SME/NEON scheduling and how the KleidiAI kernels scale with larger models and longer prompts.