[anthonymaltsev]

Overview

Reorder reshaping so that short conv block runs as a single gemm over all sequences instead of many separate gemv per sequences (reading weights each time). This affects batched decode primarily. Prefill is unchanged for quantized weights, but it's faster for F16/BF16 with several sequences, because it enables those to go through hipBLAS or cuBLAS.

Additional Information

Here are some benchmarking results from AMD MI325X and NVIDIA H100 comparing the base commit (0ce8cd1) against the branch, using llama-batched-bench for text generation (-npp 128 -ntg 128 -c 16384 -b 2048 -ub 512)

MI325X:

model	B=1	B=8	B=16	B=32	B=64
LFM2.5-1.2B Q4_0	651 → 654 (+0%)	3271 → 3470 (+6%)	5236 → 6024 (+15%)	7426 → 9634 (+30%)	9709 → 14408 (+48%)
LFM2.5-1.2B F16	496 → 501 (+1%)	3086 → 3544 (+15%)	1962 → 1962 (0%)	2163 → 2135 (-1%)	3701 → 4072 (+10%)
LFM2.5-2.6B Q4_0	401 → 410 (+2%)	1736 → 1876 (+8%)	2792 → 3285 (+18%)	3883 → 5239 (+35%)	4981 → 7905 (+59%)
LFM2.5-8B-A1B Q4_0	427 → 433 (+2%)	2049 → 2205 (+8%)	2400 → 2704 (+13%)	3392 → 4169 (+23%)	4587 → 6384 (+39%)
LFM2.5-24B-A2B Q4_0	281 → 287 (+2%)	1233 → 1332 (+8%)	1208 → 1337 (+11%)	1721 → 2042 (+19%)	2377 → 3135 (+32%)
LFM2.5-24B-A2B Q2_K	271 → 276 (+2%)	1275 → 1250 (-2%)	1143 → 1179 (+3%)	1644 → 1742 (+6%)	1860 → 1993 (+7%)

H100:

model	B=1	B=8	B=16	B=32	B=64
LFM2.5-1.2B Q4_0	974 → 979 (+0%)	4049 → 4886 (+21%)	5732 → 7567 (+32%)	7686 → 12096 (+57%)	9207 → 16890 (+83%)
LFM2.5-1.2B F16	712 → 705 (-1%)	3050 → 4070 (+33%)	4312 → 7468 (+73%)	5564 → 12657 (+127%)	6415 → 19602 (+206%)
LFM2.5-2.6B Q4_0	514 → 519 (+1%)	2045 → 2502 (+22%)	2899 → 3978 (+37%)	3869 → 6419 (+66%)	4612 → 9178 (+99%)
LFM2.5-8B-A1B Q4_0	639 → 640 (+0%)	1942 → 2276 (+17%)	2289 → 2758 (+21%)	3149 → 4275 (+36%)	3964 → 5988 (+51%)
LFM2.5-24B-A2B Q4_0	420 → 422 (+0%)	1264 → 1505 (+19%)	1548 → 1897 (+23%)	1914 → 2548 (+33%)	2324 → 3415 (+47%)
LFM2.5-24B-A2B Q4_K_M	389 → 390 (+0%)	1087 → 1291 (+19%)	1304 → 1634 (+25%)	1695 → 2355 (+39%)	1990 → 3004 (+51%)

Perplexity over wikitext remained unchanged.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, claude opus 5.5 used to write code and profile performance.

[anthonymaltsev]

@tdakhran @kbenkhaled

[ngxson]

can you check if ggml_build_forward_order works better?

[anthonymaltsev]

can you check if ggml_build_forward_order works better?

I tried this and get very close to the exact same results as the current implementation above. At batch size 1, the reordering implementation in the PR right now is better by 1-2%, and at higher batch sizes they are within noise (+- 0.2%). I profiled this on the MI325X

[ngxson]

I think we prefer ggml_build_forward_order in general, please adapt the PR accordingly

[tdakhran]

@anthonymaltsev could you please address comments?

[anthonymaltsev]

Ok double checked more carefully, and i think ggml_build_forward_order does not suffice for these changes. As I understand it, ggml_build_forward_order only reorders execution of nodes so long as data dependencies are not changed (eg ordering of computational branches), where this PR changes the order of nodes along one branch of the graph, so that the matmul operates on a 2d tensor, instead of a 3d tensor. This induces a data dependency change that is not possible to achieve with a ggml_build_forward_order change. My previous reply which suggested it was possible was actually the current reshape order swap PLUS the ggml_build_forward_order, which did nothing except defer graph node additions -- sorry for the confusion there. Can you clarify how you envisioned ggml_build_forward_order to be used @ngxson ?

This PR is similar change to #27513 and #30823.