[Xarbirus]

Overview

This reverts commit 5e03a1f.

Fixes #30704, fixes #30678

Reason for reverting the commit: significant confirmed performance degradation on CUDA and Vulkan. At the same time, on Metal, only a group of the scenarios will be broken following the revert, not all quants and uses of the model. To get NaN, the following conditions must be met:

1. actual ubatch >= 32
2. any quant other than BF16 and MXFP4
3. in ubatch, there is a token for which silu(gate) * up is greater than 65504

This can be considered a temporary measure, pending the implementation of precision in the backends (#30260 for CUDA, #30656 for Vulkan).

Additional information

Apple M2 Max (64 GB), macOS 15.8.1, Metal backend.

Perplexity (trunk only, no MTP)

llama-perplexity -f prompt -c 2048 -b 2048 -ngl 99 (default -ub 512), 3 chunks

Quant	base (master)	revert
Q8_0	2.1771 2.1360 2.3170	2.1771 2.1360 nan
Q6_K	2.2692 2.2252 2.4298	nan nan nan
Q4_K_M	2.2476 2.2383 2.5125	nan nan nan
BF16	2.2062 2.1612 2.3576	2.2062 2.1612 2.3576
MXFP4_MOE	2.2803 2.2543 2.4710	2.2803 2.2543 2.4710

MTP (speculative decoding with the MTP head)

llama-speculative-simple -ngl 99 -ngld 99 --spec-type draft-mtp --spec-draft-n-max 1 -f prompt -n 256 --temp 0 --top-k 1 --seed 42 -c 8192 -b 8192 -ub 2048

Quant	base: accept	base: t/s	revert: accept	revert: t/s
Q8_0	125/133 = 94.0%	82.6	0/257 = 0.0%	44.0
Q6_K	122/135 = 90.4%	82.3	0/257 = 0.0%	38.8
Q4_K_M	119/138 = 86.2%	93.0	119/138 = 86.2%	92.9
BF16	121/137 = 88.3%	69.0	121/137 = 88.3%	68.8
MXFP4_MOE	122/135 = 90.4%	113.6	122/135 = 90.4%	113.4

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[Green-Sky]

Should be good. At least until a variant of #30656 has been decided on.

[taronaeo]

Hmm, sorry but I am under the impression that the performance degradation fix belongs to the individual backends. Reverting this is just a short-term solution.

I can see some discussions happening surrounding it though: #30656

[Xarbirus]

@taronaeo I agree with @Green-Sky's point: the current state degrades performance for many users (CUDA, Vulkan, etc.), whereas reverting the changes would only affect a subset of users (Metal + specific quantizations + MTP + the problem input-dependent). Therefore, I suggest merging the current revert and waiting for a resolution regarding #30656.

[ggerganov]

Please test if #30861 works as expected on all backend.

[taronaeo]

Closing as #30861 appears to have fixed it.