[akapoor3518]

##########
TSISIM RESULT
root@tsisim:/usr/bin/tsi/bin# ./run_llama_cli.sh
MODEL: tinyllama-vo-5m-para.gguf
INFO: updated /usr/bin/tsi/bin/tsi-ggml/tsavorite-model-deployment.yaml with txe_count:20, multi_thread_enable:true; preserved advanced_matmul_shape_offload:true, triton_matmul_small_n_transpose_opt:true

TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=20 multi_thread_enable=1 user_dram_size_gb=16 user_dram_size_bytes=17179869184 advanced_matmul_shape_offload=1 triton_matmul_small_n_transpose_opt=1
[2026-08-05 20:44:57.832] [info] </proj/work/srv-ghr/ral/FlexHAL/src/hal.c:170> hal_lib_init: topology from tsi-soc-hal: ral_tag=SKYLP_G0741 num_chiplets=10 chiplet_stride=0x1c000000 ral_csr_base=0x20000000
[2026-08-05 20:44:57.837] [info] </proj/work/srv-ghr/ral/FlexHAL/src/hal.c:263> hal_lib_init: complete — 10 chiplets, ral_tag=SKYLP_G0741
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:823> num_vm_segments: 2
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:826> VM segment 0: start_addr = 0xffb055105000, size = 17079185408
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:833> VMA: map_addr = 0xffb055105000, map_size = 17079185408
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:835> VMA: dev_buf = 0xffb5682f03b8
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:836> VMA: dev_buf_num = 1
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:837> VMA: dev_buf_offset = 100683776
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:826> VM segment 1: start_addr = 0xffb04f100000, size = 100683776
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:833> VMA: map_addr = 0xffb04f100000, map_size = 100683776
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:835> VMA: dev_buf = 0xffb5682f0490
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:836> VMA: dev_buf_num = 2
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:837> VMA: dev_buf_offset = 0
was Tim. He loved

llama_perf_sampler_print: sampling time = 9.32 ms / 11 runs ( 0.85 ms per token, 1180.38 tokens per second)
llama_perf_context_print: load time = 2848.88 ms
llama_perf_context_print: prompt eval time = 2615.44 ms / 6 tokens ( 435.91 ms per token, 2.29 tokens per second)
llama_perf_context_print: eval time = 8887.19 ms / 4 runs ( 2221.80 ms per token, 0.45 tokens per second)
llama_perf_context_print: total time = 11749.55 ms / 10 tokens

=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD OPU 80 150 13161 164.51
MUL OPU 85 160 14798 174.09
RMS_NORM OPU 85 85 9229 108.58
MUL_MAT CPU 696 0 116249 167.02
MUL_MAT OPU 165 1700 9118989 55266.60
CONT OPU 80 0 2510 31.38
RESHAPE CPU 338 0 246 0.73
VIEW CPU 346 0 39 0.11
VIEW OPU 80 0 104 1.30
PERMUTE CPU 218 0 66 0.30
PERMUTE OPU 80 0 55 0.69
TRANSPOSE CPU 119 0 31 0.26
GET_ROWS OPU 15 0 134 8.93
SET_ROWS CPU 254 0 1082 4.26
SOFT_MAX OPU 40 0 13510 337.75
ROPE CPU 214 0 2235 10.44
GLU OPU 40 75 554598 13864.95

OPU Profiling Results:
Profiler disabled

root@tsisim:/usr/bin/tsi/bin#
root@tsisim:/usr/bin/tsi/bin#
root@tsisim:/usr/bin/tsi/bin#
root@tsisim:/usr/bin/tsi/bin# /tsi/tsi-sw/check_tsictl.sh
Index 0: execute : 8874
execute_debug : 0
Index 1: execute : 2989
execute_debug : 0
Index 2: execute : 2989
execute_debug : 0
Index 3: execute : 2989
execute_debug : 0
Index 4: execute : 2909
execute_debug : 0
Index 5: execute : 2784
execute_debug : 0
Index 6: execute : 2384
execute_debug : 0
Index 7: execute : 2384
execute_debug : 0
Index 8: execute : 2384
execute_debug : 0
Index 9: execute : 2384
execute_debug : 0
Index 10: execute : 2384
execute_debug : 0
Index 11: execute : 2384
execute_debug : 0
Index 12: execute : 2384
execute_debug : 0
Index 13: execute : 2384
execute_debug : 0
Index 14: execute : 2384
execute_debug : 0
Index 15: execute : 2384
execute_debug : 0
Index 16: execute : 2304
execute_debug : 0
Index 17: execute : 2304
execute_debug : 0
Index 18: execute : 2304
execute_debug : 0
Index 19: execute : 1162
execute_debug : 0
root@tsisim:/usr/bin/tsi/bin#

#########
POSIX RESULT
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
INFO: fixing GLIBC compatibility for TSI binaries (build-posix)
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$ build-posix/bin/llama-cli -p "My cat's name is" -m /proj/rel/sw/ggml/models/tinyllama-vo-5m-para.gguf --device tsavorite -c 12288 --temp 0.0 --n-predict 1 --repeat-penalty 1.5 -b 1024 --top-k 50 --top-p 0.9 --repeat-last-n 5 --no-warmup --no-conversation

TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=1 multi_thread_enable=1 user_dram_size_gb=8 user_dram_size_bytes=8589934592 advanced_matmul_shape_offload=1 triton_matmul_small_n_transpose_opt=1
My cat's name is Tim

llama_perf_sampler_print: sampling time = 2.22 ms / 8 runs ( 0.28 ms per token, 3598.74 tokens per second)
llama_perf_context_print: load time = 1102.06 ms
llama_perf_context_print: prompt eval time = 670.98 ms / 7 tokens ( 95.85 ms per token, 10.43 tokens per second)
llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_perf_context_print: total time = 1104.63 ms / 8 tokens

=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD OPU 16 100 25228 1576.75
MUL OPU 17 107 91481 5381.24
RMS_NORM OPU 17 17 13016 765.65
MUL_MAT CPU 123 0 32512 264.33
MUL_MAT OPU 33 33 439457 13316.88
CONT OPU 16 0 1330 83.12
RESHAPE CPU 95 0 27 0.28
VIEW CPU 83 0 11 0.13
VIEW OPU 16 0 5 0.31
PERMUTE CPU 62 0 18 0.29
PERMUTE OPU 16 0 2 0.12
TRANSPOSE CPU 27 0 6 0.22
GET_ROWS OPU 3 0 840 280.00
SET_ROWS CPU 60 0 76 1.27
SOFT_MAX OPU 8 0 7627 953.38
ROPE CPU 63 0 325 5.16
GLU OPU 8 50 37294 4661.75

OPU Profiling Results:
Profiler disabled

[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$ build-posix/bin/llama-cli -p "My cat's name is" -m /proj/rel/sw/ggml/models/Tiny-Llama-v0.3-FP32-1.1B-F32.gguf --device tsavorite -c 12288 --temp 0.0 --n-predict 1 --repeat-penalty 1.5 -b 1024 --top-k 50 --top-p 0.9 --repeat-last-n 5 --no-warmup --no-conversation

TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=1 multi_thread_enable=1 user_dram_size_gb=8 user_dram_size_bytes=8589934592 advanced_matmul_shape_offload=1 triton_matmul_small_n_transpose_opt=1
My cat's name is L

llama_perf_sampler_print: sampling time = 2.08 ms / 8 runs ( 0.26 ms per token, 3853.56 tokens per second)
llama_perf_context_print: load time = 121830.23 ms
llama_perf_context_print: prompt eval time = 107706.40 ms / 7 tokens (15386.63 ms per token, 0.06 tokens per second)
llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_perf_context_print: total time = 121834.95 ms / 8 tokens

=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD OPU 44 296 103829 2359.75
MUL OPU 45 303 213007 4733.49
RMS_NORM OPU 45 45 114679 2548.42
MUL_MAT CPU 403 0 1177178 2921.04
MUL_MAT OPU 89 89 102969597 1156961.76
CONT OPU 44 0 7373 167.57
RESHAPE CPU 237 0 217 0.92
VIEW CPU 240 0 22 0.09
VIEW OPU 44 0 12 0.27
PERMUTE CPU 157 0 77 0.49
PERMUTE OPU 44 0 8 0.18
TRANSPOSE CPU 87 0 23 0.26
GET_ROWS OPU 3 0 3204 1068.00
SET_ROWS CPU 175 0 345 1.97
SOFT_MAX OPU 22 0 41071 1866.86
ROPE CPU 174 0 7221 41.50
GLU OPU 22 148 329728 14987.64

OPU Profiling Results:
Profiler disabled

[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$ ls -lrt
total 20204
-rw-r--r-- 1 akapoor tsiusers 47860 Aug 5 10:07 AUTHORS
-rw-r--r-- 1 akapoor tsiusers 12356 Aug 5 10:07 CMakeLists.txt
-rw-r--r-- 1 akapoor tsiusers 4570 Aug 5 10:07 CMakePresets.json
-rw-r--r-- 1 akapoor tsiusers 6091 Aug 5 10:07 CODEOWNERS
-rw-r--r-- 1 akapoor tsiusers 8481 Aug 5 10:07 CONTRIBUTING.md
-rw-r--r-- 1 akapoor tsiusers 1078 Aug 5 10:07 LICENSE
-rw-r--r-- 1 akapoor tsiusers 257 Aug 5 10:07 Makefile
-rw-r--r-- 1 akapoor tsiusers 33656 Aug 5 10:07 README.md
-rw-r--r-- 1 akapoor tsiusers 5347 Aug 5 10:07 SECURITY.md
-rwxr-xr-x 1 akapoor tsiusers 21790 Aug 5 10:07 build-xcframework.sh
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 ci
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 cmake
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 common
-rwxr-xr-x 1 akapoor tsiusers 458746 Aug 5 10:07 convert_hf_to_gguf.py
-rwxr-xr-x 1 akapoor tsiusers 24353 Aug 5 10:07 convert_hf_to_gguf_update.py
-rwxr-xr-x 1 akapoor tsiusers 19106 Aug 5 10:07 convert_llama_ggml_to_gguf.py
-rwxr-xr-x 1 akapoor tsiusers 20291 Aug 5 10:07 convert_lora_to_gguf.py
drwxr-xr-x 6 akapoor tsiusers 4096 Aug 5 10:07 docs
drwxr-xr-x 28 akapoor tsiusers 4096 Aug 5 10:07 examples
-rw-r--r-- 1 akapoor tsiusers 1556 Aug 5 10:07 flake.lock
-rw-r--r-- 1 akapoor tsiusers 7243 Aug 5 10:07 flake.nix
drwxr-xr-x 5 akapoor tsiusers 4096 Aug 5 10:07 gguf-py
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 grammars
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 include
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 licenses
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 media
-rw-r--r-- 1 akapoor tsiusers 3319 Aug 5 10:07 model-rerun.py
drwxr-xr-x 3 akapoor tsiusers 4096 Aug 5 10:07 models
-rw-r--r-- 1 akapoor tsiusers 163 Aug 5 10:07 mypy.ini
drwxr-xr-x 3 akapoor tsiusers 4096 Aug 5 10:07 pocs
-rw-r--r-- 1 akapoor tsiusers 124786 Aug 5 10:07 poetry.lock
-rw-r--r-- 1 akapoor tsiusers 1336 Aug 5 10:07 pyproject.toml
-rw-r--r-- 1 akapoor tsiusers 616 Aug 5 10:07 pyrightconfig.json
-rw-r--r-- 1 akapoor tsiusers 551 Aug 5 10:07 requirements.txt
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 requirements
drwxr-xr-x 5 akapoor tsiusers 4096 Aug 5 10:07 scripts
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 tests
drwxr-xr-x 17 akapoor tsiusers 4096 Aug 5 10:07 tools
-rwxr-xr-x 1 akapoor tsiusers 48962 Aug 5 10:07 tsi-pkg-build.sh
drwxr-xr-x 7 akapoor tsiusers 4096 Aug 5 10:07 vendor
drwxr-xr-x 11 akapoor tsiusers 4096 Aug 5 10:15 ggml-tsi-kernel
drwxr-xr-x 12 akapoor tsiusers 4096 Aug 5 10:24 build-fpga
drwxr-xr-x 3 akapoor tsiusers 4096 Aug 5 10:25 tsi-ggml
-rw-r--r-- 1 akapoor tsiusers 19509080 Aug 5 10:25 tsi-ggml-0.4.21.tz
-rwxr-xr-x 1 akapoor tsiusers 993 Aug 5 10:26 tsavorite-model-deployment.yaml
-rw-r--r-- 1 akapoor tsiusers 2403 Aug 5 10:33 run.log
drwxr-xr-x 2 akapoor tsiusers 8192 Aug 5 12:22 src
-rwxr-xr-x 1 akapoor tsiusers 9515 Aug 5 12:29 patch_tsavorite_mixed_precision_matmul.sh
-rwxr-xr-x 1 akapoor tsiusers 8091 Aug 5 12:37 patch_tsavorite_mixed_precision_matmul_v2.sh
-rwxr-xr-x 1 akapoor tsiusers 7066 Aug 5 12:38 patch_tsavorite_mixed_precision_matmul_v3.sh
-rwxr-xr-x 1 akapoor tsiusers 2341 Aug 5 12:47 patch_tsavorite_matmul_bf16_support.sh
-rwxr-xr-x 1 akapoor tsiusers 1848 Aug 5 12:49 patch_tsavorite_none_bf16_support.sh
drwxr-xr-x 5 akapoor tsiusers 4096 Aug 5 12:56 ggml
drwxr-xr-x 12 akapoor tsiusers 4096 Aug 5 13:22 build-posix
-rw-r--r-- 1 akapoor tsiusers 0 Aug 5 13:24 tsi-op.txt
-rw-r--r-- 1 akapoor tsiusers 11135 Aug 5 13:26 ggml_perf-all-shape.log
-rw-r--r-- 1 akapoor tsiusers 7866 Aug 5 13:26 ggml_op_shape_dtype_catalog.tsv
[akapoor@wspd0 llama.cpp]$ cat ggml_op_shape_dtype_catalog.tsv
Op Decision Count Result Src0 Src1

---

ADD SUPPORTED 46 f32 [2048,1,1,1] f32 [2048,1,1,1] f32 [2048,1,1,1]
ADD SUPPORTED 37 f32 [2048,512,1,1] f32 [2048,512,1,1] f32 [2048,512,1,1]
ADD SUPPORTED 42 f32 [2048,7,1,1] f32 [2048,7,1,1] f32 [2048,7,1,1]
CONT SUPPORTED 22 f16 [12288,64,4,1] f16 [12288,64,4,1] none [0,0,0,0]
CONT SUPPORTED 22 f16 [256,64,4,1] f16 [256,64,4,1] none [0,0,0,0]
CONT SUPPORTED 22 f32 [2048,512,1,1] f32 [64,32,512,1] none [0,0,0,0]
CONT SUPPORTED 22 f32 [2048,7,1,1] f32 [64,32,7,1] none [0,0,0,0]
CPY SUPPORTED 1 f16 [12288,64,1,1] f32 [12288,64,1,1] f16 [12288,64,1,1]
FLASH_ATTN_EXT REJECTED 22 f32 [64,32,1,1] f32 [64,1,32,1] f16 [64,12288,4,1]
GET_ROWS SUPPORTED 2 f32 [2048,1,1,1] f32 [2048,1,1,1] i32 [1,1,1,1]
GET_ROWS SUPPORTED 1 f32 [2048,1,1,1] f32 [2048,32003,1,1] i32 [1,1,1,1]
GET_ROWS SUPPORTED 2 f32 [2048,1,1,1] f32 [2048,7,1,1] i32 [1,1,1,1]
GET_ROWS SUPPORTED 2 f32 [2048,512,1,1] f32 [2048,512,1,1] i32 [512,1,1,1]
GET_ROWS SUPPORTED 1 f32 [2048,7,1,1] f32 [2048,32003,1,1] i32 [7,1,1,1]
GLU SUPPORTED 23 f32 [5632,1,1,1] f32 [5632,1,1,1] f32 [5632,1,1,1]
GLU SUPPORTED 17 f32 [5632,512,1,1] f32 [5632,512,1,1] f32 [5632,512,1,1]
GLU SUPPORTED 21 f32 [5632,7,1,1] f32 [5632,7,1,1] f32 [5632,7,1,1]
MUL_MAT REJECTED 22 f32 [12288,512,32,1] f16 [64,12288,4,1] f32 [64,512,32,1]
MUL_MAT REJECTED 22 f32 [256,7,32,1] f16 [64,256,4,1] f32 [64,7,32,1]
MUL_MAT REJECTED 22 f32 [64,512,32,1] f16 [12288,64,4,1] f32 [12288,512,32,1]
MUL_MAT REJECTED 22 f32 [64,7,32,1] f16 [256,64,4,1] f32 [256,7,32,1]
MUL_MAT SUPPORTED 44 f32 [2048,1,1,1] f32 [2048,2048,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 23 f32 [2048,1,1,1] f32 [5632,2048,1,1] f32 [5632,1,1,1]
MUL_MAT SUPPORTED 40 f32 [2048,512,1,1] f32 [2048,2048,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 19 f32 [2048,512,1,1] f32 [5632,2048,1,1] f32 [5632,512,1,1]
MUL_MAT SUPPORTED 44 f32 [2048,7,1,1] f32 [2048,2048,1,1] f32 [2048,7,1,1]
MUL_MAT SUPPORTED 21 f32 [2048,7,1,1] f32 [5632,2048,1,1] f32 [5632,7,1,1]
MUL_MAT SUPPORTED 44 f32 [256,1,1,1] f32 [2048,256,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 37 f32 [256,512,1,1] f32 [2048,256,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 44 f32 [256,7,1,1] f32 [2048,256,1,1] f32 [2048,7,1,1]
MUL_MAT SUPPORTED 2 f32 [32003,1,1,1] f32 [2048,32003,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 2 f32 [32003,512,1,1] f32 [2048,32003,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 46 f32 [5632,1,1,1] f32 [2048,5632,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 37 f32 [5632,512,1,1] f32 [2048,5632,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 42 f32 [5632,7,1,1] f32 [2048,5632,1,1] f32 [2048,7,1,1]
MUL SUPPORTED 52 f32 [2048,1,1,1] f32 [2048,1,1,1] f32 [2048,1,1,1]
MUL SUPPORTED 34 f32 [2048,512,1,1] f32 [2048,512,1,1] f32 [2048,1,1,1]
MUL SUPPORTED 43 f32 [2048,7,1,1] f32 [2048,7,1,1] f32 [2048,1,1,1]
NONE SUPPORTED 45 f32 [2048,1,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 44 f32 [2048,2048,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 44 f32 [2048,256,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 1 f32 [2048,32003,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 44 f32 [2048,5632,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 22 f32 [5632,2048,1,1] none [0,0,0,0] none [0,0,0,0]
RMS_NORM SUPPORTED 47 f32 [2048,1,1,1] f32 [2048,1,1,1] none [0,0,0,0]
RMS_NORM SUPPORTED 37 f32 [2048,512,1,1] f32 [2048,512,1,1] none [0,0,0,0]
RMS_NORM SUPPORTED 43 f32 [2048,7,1,1] f32 [2048,7,1,1] none [0,0,0,0]
ROPE SUPPORTED 22 f32 [64,32,1,1] f32 [64,32,1,1] i32 [1,1,1,1]
ROPE SUPPORTED 22 f32 [64,32,512,1] f32 [64,32,512,1] i32 [512,1,1,1]
ROPE SUPPORTED 22 f32 [64,32,7,1] f32 [64,32,7,1] i32 [7,1,1,1]
ROPE SUPPORTED 22 f32 [64,4,1,1] f32 [64,4,1,1] i32 [1,1,1,1]
ROPE SUPPORTED 21 f32 [64,4,512,1] f32 [64,4,512,1] i32 [512,1,1,1]
ROPE SUPPORTED 22 f32 [64,4,7,1] f32 [64,4,7,1] i32 [7,1,1,1]
SOFT_MAX SUPPORTED 22 f32 [12288,512,32,1] f32 [12288,512,32,1] f32 [12288,512,1,1]
SOFT_MAX SUPPORTED 22 f32 [256,7,32,1] f32 [256,7,32,1] f32 [256,64,1,1]
[akapoor@wspd0 llama.cpp]$

---

Summary by cubic

Enables generic mixed‑precision MAT_MUL offload by packing GGML inputs to F32 for the Triton TMU; outputs remain F32. Addresses FIR‑2152 by extending offload to F16, BF16, and quantized weights (block‑aligned, contiguous K) with tighter checks and clearer errors.

• New Features

  • Offloads when A/B are F32/F16/BF16 or quantized types with to_float, if K is block‑aligned and nb0 matches the type size; result tensor stays F32.
  • Adds unified, K‑stride aware F32 pack and scatter helpers for both full and small‑N transpose paths, handling non‑contiguous strides and quantized blocks.
  • Improves capability cataloging: marks GGML_OP_NONE tensors with packable dtypes as SUPPORTED and records precise reasons (e.g., mul_mat shape/dtype supported or not).

• Refactors

  • Replaces manual A/B copy loops with centralized helpers, validates nb0 vs. type size, and adds explicit error messages with tsi_cleanup() before abort.
  • Adds comments clarifying host‑side dequantization via GGML type traits before calling the F32 Triton MAT_MUL kernel.

^{Written for commit 33641ee. Summary will update on new commits.}

[qodo-code-review]

PR Summary by Qodo

Enable mixed-precision MUL_MAT offload via host packing to F32

✨ Enhancement 🕐 40+ Minutes

AI Description

• Allow Triton TMU MUL_MAT offload when inputs are convertible to F32.
• Add robust host-side packing helpers for F32/F16/BF16 and quantized types.
• Record support/rejection reasons in the op shape+dtype catalog for easier debugging.

Diagram

graph TD
  A["GGML op: MUL_MAT"] --> B{"Supported?"} --> C["Pack inputs to F32"] --> D["Triton TMU MAT_MUL"] --> E[("F32 output")]
  C --> F[("GGML type traits")]
  B --> G["Catalog record"]

  subgraph Legend
    direction LR
    _op["Op/Function"] ~~~ _dec{"Decision"} ~~~ _db[("Metadata/Traits")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Native mixed-dtype Triton kernels (F16/BF16 inputs)

• ➕ Avoids host-side conversion cost and extra temporary buffers
• ➕ Potentially higher throughput and lower memory bandwidth pressure
• ➖ Requires additional kernel variants and validation matrix
• ➖ More complex dtype handling (accumulation type, numerics)

2. Cache prepacked F32 buffers across repeated MUL_MAT ops

• ➕ Reduces repeated pack cost for reused weights/activations
• ➕ Can significantly improve end-to-end latency for iterative inference
• ➖ Needs lifecycle management and invalidation when tensors change
• ➖ Increases memory footprint and introduces cache complexity

3. Restrict support to a curated set of dtypes (F16/BF16/F32 only)

• ➕ Simpler correctness surface area than generic trait-based conversion
• ➕ Easier performance predictability
• ➖ Blocks offload for quantized types that could be converted safely
• ➖ Less flexible for model variants and future GGML types

Recommendation: Current approach (accept any GGML dtype that can be converted to F32 during packing, while keeping F32 output) is a pragmatic first step that expands offload coverage with limited kernel changes. Consider adding optional caching of packed buffers next if profiling shows packing dominates, and only pursue native mixed-dtype kernels if packing cost becomes a persistent bottleneck.

Files changed (1) +155 / -27

Enhancement (1) +155 / -27

ggml-tsavorite.cppGeneralize Triton MUL_MAT packing to accept any F32-convertible input type +155/-27

Generalize Triton MUL_MAT packing to accept any F32-convertible input type

• Adds helpers to detect whether a ggml type can be packed to F32 and to copy/convert K elements into an F32 buffer (handling F32/F16/BF16 directly and other types via ggml type traits). Updates MUL_MAT eligibility checks to allow mixed-precision inputs (convertible to F32) while keeping the output F32, and routes all packing paths through the new conversion helper for consistent behavior and clearer error reporting.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Quant block size unchecked ✓ Resolved 🐞 Bug ≡ Correctness

Description

tsavorite_tensor_copy_k_to_f32() calls traits->to_float(base, dst, K) for quantized types without
validating that K is a multiple of ggml_blck_size(t->type), which many quantized dequantizers
assume; this can yield incorrect data or out-of-bounds reads. The PR introduces this path by
allowing any type with traits->to_float in MAT_MUL packing.

Code

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[R3020-3022]

+
+        traits->to_float(base, dst, K);
+        return;

Relevance

●●● Strong

Team often adds/keeps strict MAT_MUL alignment asserts; block-size guard likely welcomed for
correctness.

PR-#131
PR-#150

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new packing helper directly calls traits->to_float(base, dst, K) without checking the quant
block-size alignment requirement that ggml itself treats as invariant for row sizing, and other
backends assert before quant copies.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[2969-3023]
ggml/src/ggml.c[1248-1251]
ggml/src/ggml-metal/ggml-metal-ops.cpp[1351-1354]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`tsavorite_tensor_copy_k_to_f32()` invokes `traits->to_float(base, dst, K)` for quantized inputs but does not enforce the fundamental precondition that `K` is block-aligned for that quant type. This can cause incorrect conversion and, depending on the dequantizer implementation, out-of-bounds reads.

### Issue Context
In ggml, quantized row sizes require `ne % ggml_blck_size(type) == 0` (see `ggml_row_size()`), and other backends assert this before operating on quant data.

### Fix Focus Areas
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[3007-3022]

### Suggested fix
- In the `traits && traits->to_float` branch, compute `const int64_t bs = ggml_blck_size(t->type);` and hard-fail (or return a failure to allow fallback) if `bs <= 0 || (K % bs) != 0`.
- (Optional but recommended) also enforce the same `K % bs == 0` constraint in `mul_mat_supported_size()` when accepting quantized types so unsupported shapes are rejected before execution.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Quant stride may abort ✓ Resolved 🐞 Bug ☼ Reliability

Description

mul_mat_supported_size() now accepts quantized inputs based on “type has traits->to_float”, but the
runtime packer aborts when a quant tensor’s K-stride nb0 differs from ggml_type_size(type). This
creates a supports_op/execution mismatch where an op can be accepted and then hard-crash during
packing for view/non-contiguous tensors.

Code

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[R3047-3052]

+    const bool a_dtype_ok = tsavorite_tensor_type_can_pack_to_f32(a->type);
+    const bool b_dtype_ok = tsavorite_tensor_type_can_pack_to_f32(b->type);
+    const bool op_dtype_ok = (op->type == GGML_TYPE_F32);
+
+    if (!a_dtype_ok || !b_dtype_ok || !op_dtype_ok) {
#if TRITON_DEBUG

Relevance

●●● Strong

They’ve accepted hardening MAT_MUL support/dispatch invariants to avoid runtime aborts on
“supported” paths.

PR-#150
PR-#131

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The support predicate only checks that the types are convertible to F32, while the packing helper
will abort for quantized inputs when nb0 does not match ggml_type_size(type), creating a clear
accept-then-crash path.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[3042-3063]
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[3007-3019]
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[5124-5133]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`mul_mat_supported_size()` was broadened to accept any input type that can be packed to F32, but it does not validate layout/stride requirements that the packing code enforces by `abort()`. Specifically, `tsavorite_tensor_copy_k_to_f32()` requires `nb0 == ggml_type_size(type)` for quantized types and aborts otherwise.

### Issue Context
A tensor may be accepted as supported (and thus scheduled/offloaded) even if it is a view or otherwise non-standard stride in dimension 0, leading to process termination instead of a clean rejection/fallback.

### Fix Focus Areas
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[3042-3062]
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[3007-3019]

### Suggested fix
- In `mul_mat_supported_size()`, when `ggml_get_type_traits(type)->to_float` is used for acceptance (i.e., quantized / block types), also require:
 - `a->nb[0] != 0 ? a->nb[0] : ggml_type_size(a->type)` equals `ggml_type_size(a->type)` (same for `b`), OR use `ggml_is_contiguous_rows(a)` / `ggml_is_contiguous_rows(b)` if available.
 - `K % ggml_blck_size(type) == 0` (see related finding).
- Alternatively (or additionally), refactor `tsavorite_tensor_copy_k_to_f32()` to return `bool` and propagate failure up so the backend can fall back instead of aborting.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Repeated temp buffer allocation ✓ Resolved 🐞 Bug ➹ Performance

Description

The packing path now constructs std::vector<float> tmp_k(K) during B packing (single-TXE) and again
inside each multi-TXE worker, adding repeated heap allocation and zero-initialization overhead on a
hot path. This can increase latency especially for many small matmuls or high D2/D3 repetition.

Code

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[R5141-5144]

                if (use_small_n_transpose) {
+                    std::vector<float> tmp_k((size_t)K);
+
                    for (int64_t c = 0; c < M; ++c) {

Relevance

●● Moderate

Perf micro-optimization; no clear precedent on avoiding per-call std::vector allocations in this
path.

PR-#144

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The diff introduces std::vector<float> tmp_k((size_t)K) in both the single-TXE
transpose/no-transpose packing branches and in the multi-TXE worker, which repeats across
executions.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[5141-5163]
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[5340-5348]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`tmp_k` is allocated as `std::vector<float>(K)` in the MAT_MUL packing path. While allocated once per packing invocation (not per inner-loop iteration), it still repeats for each `(d2,d3)` call and per multi-TXE worker, causing extra allocations and initialization.

### Issue Context
This code runs on the critical path prior to each Triton launch and can be optimized by reusing scratch space.

### Fix Focus Areas
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[5141-5163]
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[5340-5348]

### Suggested fix
- Hoist `tmp_k` outside the branch and reuse it: `std::vector<float> tmp_k; tmp_k.resize((size_t)K);` per `(d2,d3)` iteration.
- For the multi-TXE lambda, use a `thread_local std::vector<float>` scratch (or a per-device scratch allocated alongside `A_tile/B_tile/C_tile`) and `resize(K)` each call.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context used

✅ Compliance rules (platform): 6 rules

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

[cubic-dev-ai]

All reported issues were addressed across 1 file

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[atrivedi-tsavoritesi]

Did you run any FP16 of BF16 model ? If you run qwen:0.5b or Gemma3:270M they are quantized to q8 but will be good to run a FP16 or BF16 model as well to ensure no interesting issues show up.

[akapoor3518]

model-quantized.log.rtf

Did you run any FP16 of BF16 model ? If you run qwen:0.5b or Gemma3:270M they are quantized to q8 but will be good to run a FP16 or BF16 model as well to ensure no interesting issues show up.

ok i just ran following model and its geting offloaded and giving correct prompt repsonse. I have attached the log for below model
gemma3:1b-it-fp16.gguf
gemma3:1b-it-q4_K_M