[w1049]

Overview

This PR allows the Hexagon backend to handle a safe subset of quantized MUL_MAT cases with broadcasted src1 dimensions by flattening contiguous rows into a single 2D matrix. The existing HVX 2D matmul kernel already handled this layout, but the backend previously rejected these shapes, causing operations such as Qwen3.5's multi-sequence ssm_out projection to fall back to the CPU. It also updates the HMX path to support the flattened layout.

Fixes #26094

Additional information

Performance

Performance results for Qwen3.5-2B-Q4_0 using llama-batched-bench:

HTP	NPL	Overall master (t/s)	Overall this-pr (t/s)	Change
v73	1	83.55 ± 0.30	83.21 ± 0.28	−0.4%
v73	2	32.37 ± 0.13	52.29 ± 0.24	+61.5%
v73	4	46.08 ± 0.36	74.72 ± 2.40	+62.2%
v73	8	60.21 ± 0.50	97.42 ± 1.66	+61.8%
v75	1	87.99 ± 0.75	87.87 ± 0.44	−0.1%
v75	2	27.42 ± 0.20	48.27 ± 0.43	+76.0%
v75	4	38.41 ± 0.25	65.79 ± 0.60	+71.3%
v75	8	54.16 ± 0.41	86.49 ± 1.45	+59.7%

PP512 details

HTP	NPL	master (t/s)	this-pr (t/s)	Change
v73	1	460.49 ± 2.92	461.72 ± 1.63	+0.3%
v73	2	112.76 ± 0.52	133.29 ± 0.60	+18.2%
v73	4	129.77 ± 3.60	147.86 ± 7.88	+13.9%
v73	8	131.82 ± 0.57	152.66 ± 4.69	+15.8%
v75	1	423.51 ± 1.52	425.10 ± 2.33	+0.4%
v75	2	114.92 ± 0.60	137.19 ± 0.38	+19.4%
v75	4	112.91 ± 0.67	141.68 ± 5.36	+25.5%
v75	8	112.82 ± 2.13	139.03 ± 3.93	+23.2%

TG64 details

HTP	NPL	master (t/s)	this-pr (t/s)	Change
v73	1	11.07 ± 0.03	11.01 ± 0.05	−0.5%
v73	2	4.83 ± 0.02	8.92 ± 0.06	+84.7%
v73	4	7.48 ± 0.05	15.08 ± 0.23	+101.6%
v73	8	11.26 ± 0.19	25.02 ± 0.03	+122.2%
v75	1	11.99 ± 0.13	11.96 ± 0.07	−0.3%
v75	2	3.87 ± 0.03	7.80 ± 0.10	+101.6%
v75	4	6.12 ± 0.04	12.46 ± 0.15	+103.6%
v75	8	10.50 ± 0.01	21.50 ± 0.06	+104.8%

Graph splits

By keeping more operators on the HTP, the number of graph splits is reduced:
(but still a large number)

Details

NPL	PP512 master	PP512 this-pr	TG64 master	TG64 this-pr
1	3	3	75	75
2	51	15	123	87
4	51	15	123	87
8	51	15	123	87

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, Codex for code investigation and testing

[jhen0409]

I noticed the issue too where I'm making Q5_K support and testing https://huggingface.co/unsloth/Qwen3.5-4B-GGUF, the ssm_out projection will fallback to CPU on npl > 1. The Q5_K support (ssm_out is Q5_K in the Unsloth Q4_0 model) will slow it down more if this issue no resolved.

@max-krasnyansky do you agree with this PR? @w1049 need rebase again.

I had already verified it using the same method before, but I found this PR and the issue, so I didn't open a new PR.

[w1049]

I noticed the issue too where I'm making Q5_K support and testing https://huggingface.co/unsloth/Qwen3.5-4B-GGUF, the ssm_out projection will fallback to CPU on npl > 1. The Q5_K support (ssm_out is Q5_K in the Unsloth Q4_0 model) will slow it down more if this issue no resolved.

@max-krasnyansky do you agree with this PR? @w1049 need rebase again.

I had already verified it using the same method before, but I found this PR and the issue, so I didn't open a new PR.

Yeah, the current approach is probably a bit special-cased. I’m wondering if the better solution is to handle broadcasting properly.

[max-krasnyansky]

@w1049 please check out the MUL_MAT updates in the master.
I ended up removing all legacy paths to properly support 64-bit DMA. We have a chunked version of the HVX matmul now and all 4D cases are now routed via chunked 2D.