[dfriehs]

Overview

Tesla P100's get a nice (~20%) pp speedup by using mmq even without native dp4a. For -sm tensor configurations with multiple cards, this increases to around a +100% pp speedup (for 2 cards, possibly more with more cards).

This PR is similar to, but inverse of, it's inspiration: #24588, where gfx900 had mmq always turned on even without dp4a, and kept mmq for MoE. sm_60 will now use mmq, but only for MoE.

This might be relevant for even older NVidia hardware as well, but I only have access to a 4GB Quadro that might not be a good reference point (and no models that come to mind). For now it's gated to only the P100.

Additional information

Tests are on two P100 cards (powerlimited to 150W, connected via PCIe 3.0 x8). See comments below for tests for each quantization type (exps quantized to type tested, all other tensors Q4_0).

I tested single card performance for quantization types that fit on one card, and the speed-up closely matches -sm layer improvements.

original test, prefer results in comments below

`GGML_CUDA_P2P=1 llama-batched-bench -dev CUDA0,CUDA1 -sm tensor -no-kvu -ngl all -fa on -fit off -ub 2048 -ctk q8_0 -ctv q8_0 -npp 0,128,512,1024,2048,4096,32768,65536 -ntg 128 -npl 1 -m Qwen3.6-35B-A3B-Q4.gguf`

master:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.887	67.82	1.887	67.82
128	128	1	256	0.727	176.15	1.861	68.80	2.587	98.95
512	128	1	640	1.769	289.43	1.877	68.21	3.646	175.56
1024	128	1	1152	2.684	381.54	1.875	68.27	4.559	252.70
2048	128	1	2176	4.282	478.30	1.878	68.14	6.160	353.23
4096	128	1	4224	8.592	476.71	1.898	67.46	10.490	402.68
32768	128	1	32896	79.078	414.38	2.228	57.46	81.305	404.60
65536	128	1	65664	176.232	371.87	2.605	49.14	178.837	367.17

this pr:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.908	67.08	1.908	67.08
128	128	1	256	0.359	356.78	1.869	68.48	2.228	114.91
512	128	1	640	0.645	794.20	1.887	67.84	2.531	252.82
1024	128	1	1152	0.983	1041.43	1.889	67.76	2.872	401.09
2048	128	1	2176	1.660	1234.10	1.889	67.77	3.548	613.25
4096	128	1	4224	3.352	1222.08	1.906	67.16	5.258	803.42
32768	128	1	32896	34.815	941.19	2.238	57.21	37.053	887.81
65536	128	1	65664	88.220	742.87	2.614	48.97	90.834	722.90

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: No.

[am17an]

I have not tested this since I don't have a Pascal device but it looks to okay to me

[mb8565]

I tested this PR and recorded the prefill results on 2x Tesla P100, stock master 0278d8362, this PR's change behind an env flag so the same binary does both arms. Qwen3.6-35B-A3B-UD-Q4_K_XL, GGML_CUDA_P2P=1, llama-bench -p 2048 -r 3.

config	MMQ off	MMQ on	delta
stock default	365.52	219.18	-40%
-ub 2048 alone	521.24	301.55	-42%
-fa on + q8_0 KV	367.94	211.67	-42%
your combo minus -sm tensor	526.83	293.44	-44%
-sm tensor + -fa on	247.40	295.99	+20%
your combo (with -sm tensor)	370.93	539.36	+45%

Best config vs best config, on a 5-repeat re-run: +2.3%.

I used llama-bench rather than llama-batched-bench -fit off -no-kvu, and UD-Q4_K_XL rather than plain Q4. I reproduced the sign of your result, not its magnitude.

The -sm dependence seemed worth flagging.

[dfriehs]

Yeah, reproducing with Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf I'm getting horrible numbers in comparison as well. I can only guess that the difference is because the P100 can't handle some datatypes for mmq that well; my benchmarks above used a custom quant with IQ4_NL for all experts, while the UD quant seems to use Q6_K, Q5_K and Q4_K.

I guess I'll convert this to a draft until I can get a closer look at which quants make sense for the P100 to support for mmq.

[dfriehs]

I ran some tests with exps quantized to most types, and there are some that get a lot worse with mmq, with others more than doubling. For most types, both -sm layer/tensor improve, although layer benefits less. I don't think the small improvement for -sm tensor + mmq for select k-quants is enough to special case them. Also exciting is that with mmq enabled, -sm tensor has both better tg and pp than -sm layer for at least Qwen 3.6 35B-A3B.

quant	t/s layer	t/s tensor	t/s layer mmq	t/s tensor mmq
Q1_0	628.56	514.98	757.83	1261.90
Q2_0	573.81	477.82	737.18	1238.31
Q4_0	583.74	477.29	722.34	1219.73
Q4_1	607.81	478.50	713.18	1205.11
Q5_0	560.34	464.43	713.48	1198.58
Q5_1	580.79	463.78	709.72	1196.90
IQ1_S	582.21	466.24	717.19	1213.75
IQ1_M	551.95	459.62	550.09	456.44
IQ2_XXS	551.02	425.54	725.02	1214.13
IQ2_XS	566.18	454.69	713.14	1198.41
IQ2_S	565.08	450.11	710.47	1191.67
IQ3_XXS	563.41	449.61	717.54	1203.85
IQ3_S	565.07	448.96	712.77	1195.57
IQ4_XS	597.03	471.74	725.17	1211.36
IQ4_NL	605.17	446.09	727.28	1217.19
Q2_K	608.29	479.73	225.35	422.11
Q3_K	588.45	464.43	684.71	1153.93
Q4_K	563.45	465.14	324.63	573.27
Q5_K	591.53	469.16	272.03	499.24
Q6_K	586.72	480.66	307.83	565.17

scripts used for testing

_quants=(
  Q1_0 Q2_0 Q4_0 Q4_1 Q5_0 Q5_1
  IQ1_S IQ1_M IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_XS IQ4_NL
  Q2_K Q3_K Q4_K Q5_K Q6_K
)

for q in "${_quants[@]}"
do
  llama-quantize --tensor-type "exps=$q" --tensor-type "blk|embd|out=q4_0" --imatrix Qwen3.6-35B-A3B-imatrix.gguf Qwen3.6-35B-A3B-BF16.gguf "qwen35moe-$q.gguf" "$q"
done

export CUDA_VISIBLE_DEVICES=0,1

_quants=(
  Q1_0 Q2_0 Q4_0 Q4_1 Q5_0 Q5_1
  IQ1_S IQ1_M IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_XS IQ4_NL
  Q2_K Q3_K Q4_K Q5_K Q6_K
)

for q in "${_quants[@]}"
do
  llama-bench -m "qwen35moe-$q.gguf" -lm dio -ngl 999 -ub 2048 -p 2048 -n 32 -sm layer,tensor
done

I'm planning to switch on quant type for the P100 in ggml_cuda_should_use_mmq for now (will write the code and push later today), but it might be interesting to check out why exactly some of the quants get so much worse in the future. Thank you for testing, @mb8565.

[dfriehs]

I've finished testing all quants that support mmq right now, here's the last few ones:

quant	t/s layer	t/s tensor	t/s layer mmq	t/s tensor mmq
Q8_0 *	463.80	274.58	517.94	382.85
Q8_0 **	644.95	291.30	750.07	1523.54
MXFP4	612.46	476.98	731.80	1228.07
NVFP4	543.62	399.80	726.24	1217.29

*: -ngl 32
**: 4 Tesla P100 instead of 2

Apart from IQ1_M, which has no mmq (is this an oversight?), the only quants that do not benefit from mmq are Q2_K, Q4_K, Q5_K and Q6_K. Those quants will not use mmq on the P100 right now, but there's a TODO in there to check what the issue might be.

[dfriehs]

@am17an would it be alright to ask for a re-review? If you need me to run more tests I can do so as well.

I'm planning to look into tuning sm_60 mmq configs in a future PR unless you want it part of this one. K-quant changes will need to be a separate PR either way unless the slowdowns are just disastrous tuning.

[am17an]

I don't have hardware to test this, so I would rely on other people's reports on this patch.

[Kenzu]

I did a small test with Ornith-1.0-35B-UD-IQ4_NL.gguf, My pp stats in llama-server went from ~300 (layer, tensor was ~200) to ~900 (tensor). Token/s is close to the same 50-60. Very happy with this. But it did nothing on Q5_XL.
This is on a HP Z440 Xeon 2695v3 + 2x P100 16GB.

[dfriehs]

From what I could see the slowdown was indeed just terrible tuning causing lots of spilling on the P100. I duplicated the mmq config (-pascal-dp4a, -pascal-older) and set occupancy to 1 for all previously affected K-quants. Speeds for Q2/4/5/6_K now are comparable to previous Q3_K.

@mb8565 , @Kenzu could I ask the both of you to retest with whatever models you were testing before, including the ones you didn't get any speed-ups with? You should get a small speed-up on -sm layer or a bigger one with -sm tensor on the K-quants as well now.

I've been meaning to do a full tune of the mmq config, but the first results I got regressed for MoE so I'll have to do all of it again.

[IMbackK]

looks fine to me, but cant test either

[mb8565]

Retested on 2x Tesla P100 at master ca3d5a3e1. The retune works: every config I re-ran is a win, including the plain -sm layer default that was -40% for me in July.

model	-sm	mmq on	mmq off	delta
Qwen3.6-35B-A3B-UD-Q4_K_XL	tensor	733.33	256.47	+185.9%
Mellum2-12B-A2.5B-Q4_K_M	tensor	1128.87	578.20	+95.2%
Qwen3.6-35B-A3B-UD-Q4_K_XL	layer	624.30	366.99	+70.1%
DeepSeek-V2-Lite-Q4_K_M-REAP48	layer	1125.99	727.68	+54.7%
Mellum2-12B-A2.5B-Q4_K_M	layer	1106.96	733.56	+50.9%

llama-bench -p 2048 -n 0 -r 3 -ngl 99, CUDA_VISIBLE_DEVICES=0,1 GGML_CUDA_P2P=1, one binary with an env toggle on the sm_60 branch of ggml_cuda_should_use_mmq, arms alternated over three rounds in one session, each cell the mean of those rounds. Worst per-arm sd was 0.63%.

You were right in July about the UD quant. Its ffn_down_exps are 36 Q5_K and 4 Q6_K and gate/up are Q4_K, so this does exercise your Q4_K, Q5_K and Q6_K changes. It does not touch Q2_K or Q3_K. The other two models have Q5_0/Q8_0 ffn_down_exps.

I only re-ran the stock default. -ub 2048, -fa on + q8_0 KV, and the combo without -sm tensor are still untested, and those were three of my four July regressions. Best against best on Qwen is 733 vs 367, +99.8%, but 367 is only the best of what I ran: -ub 2048 put my July mmq-off baseline at 527, so 99.8% is an upper bound.

DeepSeek-V2-Lite-REAP48 does not load under -sm tensor on either arm. My July numbers were master 0278d8362, 567 commits back, so please don't diff them against these.

[mb8565]

Correcting the +99.8% from my last comment. Best against best on Qwen is +38.8%, which is 733.33 / 528.37.

Those two runs were not the same setup. Four things were different. 733.33 is the stock default with -sm tensor, so ubatch 512, f16 KV, and -fa left on auto. 528.37 is -sm layer -ub 2048 -fa on with q8_0 KV. The only -sm tensor mmq-off number I have for Qwen on this build is the stock default, 256.47.

Here are the three configs I skipped. All -sm layer, all on Qwen3.6-35B-A3B-UD-Q4_K_XL.

config	mmq on	mmq off	delta	July
-ub 2048	728.53	525.32	+38.7%	-42%
-fa on + q8_0 KV	617.04	365.88	+68.6%	-42%
-ub 2048 -fa on + q8_0 KV	727.28	528.37	+37.6%	-44%