Eval bug: errors and poor performance on vulkan tensor split on quad rx580 #27334

Name and Version

hi im on Linux im not sure if this place is intended to report this or not, im sorry if im reporting this to wrong place im new to git hub if you tell me ill move this text to other parts

also sorry because of my english im not native ,

also please know that im ready to cowork to fix this

on vulkan (ubuntu 26)

the default mode works ok (30~36 token/s on qwen 3.6 36b a3b) (process is 200/s)
but on tensor split (-sm split) it works at 8 toks /s ! (process is 150/s)

hardware:

4x rx580 (all same bios / okay clocks / 1100mhz, 2ghz memory)
mobo: gigabyte x99 p sli
16gb ram 1866mhz (capped by cpu)
modded bios , ReBarUEFI + NVRAM ragister 0x495 set to 0x01 using modGrubSetupvar to enable above 4G decode, also in UEFI mode and pcie 2 is display out )
cpu 2620 v3 intel

above 4g enabled also resizable bar , bar size is 8gb , also 4 cards are working in x8 (maybe one in x16?) also no igpu and everything is on 2nd gpu, i know gtt is empty (20mb) exept gpu 2 that is 1gb used

cpp is build 10452

Operating systems

Linux

GGML backends

Vulkan

Hardware

4x rx580 (all same bios / okay clocks / 1100mhz, 2ghz memory)
mobo: gigabyte x99 p sli
16gb ram 1866mhz (capped by cpu)
modded bios , ReBarUEFI + NVRAM ragister 0x495 set to 0x01 using modGrubSetupvar to enable above 4G decode, also in UEFI mode and pcie 2 is display out )
cpu 2620 v3 intel

Models

Qwen 3.6 35B A3B Q4 KXL also Q5S Unsloth also Hermes3.6-35B-A3B-Uncensored-Genesis-V7-MTP-APEX-Compact.gguf

Problem description & steps to reproduce

on vulkan (ubuntu 26)

the default mode works ok (30~36 token/s on qwen 3.6 36b a3b) (process is 200/s)
but on tensor split (-sm split) it works at 8 toks /s ! (process is 150/s)

First Bad Commit

No response

Relevant log output

aryan@hope:~/Desktop/AI$ systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ./llama-server -m ./best/1-hermes/Hermes3.6-35B-A3B-Uncensored-Genesis-V7-MTP-APEX-Compact.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b -ctk q8_0 -ctv q8_0 --tools all -fa on -sm tensor
Running as unit: run-p17763-i34208.scope; invocation ID: fc24f5a9d0ec4b689b81b74d5b5c738f
0.00.082.847 W server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)
0.00.082.986 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.085.722 W srv llama_server: -----------------
0.00.085.726 W srv llama_server: the following feature(s) are enabled:
0.00.085.726 W srv llama_server: built-in tools (experimental)
0.00.085.727 W srv llama_server: do not expose the server to untrusted environments
0.00.085.727 W srv llama_server: -----------------
0.00.088.142 I srv load_model: loading model './best/1-hermes/Hermes3.6-35B-A3B-Uncensored-Genesis-V7-MTP-APEX-Compact.gguf'
0.00.088.982 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
0.00.783.443 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
0.00.783.453 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
0.00.783.464 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring
0.00.783.468 W model has unused tensor blk.40.attn_k.weight (size = 1114112 bytes) -- ignoring
0.00.783.472 W model has unused tensor blk.40.attn_v.weight (size = 1114112 bytes) -- ignoring
0.00.783.486 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring
0.00.783.491 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.00.783.495 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.00.783.499 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 1048576 bytes) -- ignoring
0.00.783.504 W model has unused tensor blk.40.ffn_down_exps.weight (size = 285212672 bytes) -- ignoring
0.00.783.512 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 285212672 bytes) -- ignoring
0.00.783.517 W model has unused tensor blk.40.ffn_up_exps.weight (size = 285212672 bytes) -- ignoring
0.00.783.521 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 4096 bytes) -- ignoring
0.00.783.526 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring
0.00.783.530 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring
0.00.783.535 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring
0.00.783.552 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring
0.00.783.557 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
0.00.783.562 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
0.00.783.577 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring
1.57.631.162 I cmn init: llama threadpool init, n_threads = 6
1.58.270.961 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
1.58.298.413 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
1.58.299.013 I srv llama_server: model loaded
1.58.299.031 I srv llama_server: listening on http://127.0.0.1:8080
1.58.299.032 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release
1.58.299.033 W srv llama_server: ref: #26508
2.37.544.569 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
2.37.546.329 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
2.45.944.436 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 1223, progress = 0.70, t = 8.40 s / 145.65 tokens per second
2.48.737.803 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 1727, progress = 0.99, t = 11.19 s / 154.33 tokens per second
2.49.164.085 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 1735, progress = 1.00, t = 11.62 s / 149.36 tokens per second
2.54.657.825 I slot print_timing: id 3 | task 0 | prompt eval time = 11984.29 ms / 1739 tokens ( 6.89 ms per token, 145.11 tokens per second)
2.54.657.832 I slot print_timing: id 3 | task 0 | eval time = 5125.75 ms / 40 tokens ( 131.43 ms per token, 7.61 tokens per second)
2.54.657.834 I slot print_timing: id 3 | task 0 | total time = 17110.04 ms / 1779 tokens
2.54.657.841 I slot print_timing: id 3 | task 0 | graphs reused = 39
2.54.694.117 I slot release: id 3 | task 0 | stop processing: n_tokens = 1778, truncated = 0

also we have this when running qwen 27b

aryan@hope:~/Desktop/AI/best$ systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ../llama-server -m ./Qwen3.6-27B-UD-Q2_K_XL.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b -ctk q8_0 -ctv q8_0 --tools all -fa on -sm tensor --spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8
Running as unit: run-p59395-i50640.scope; invocation ID: 290d6f0d3cd2482fb52680dc4eedaae3
0.00.177.897 W server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)
0.00.178.053 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.184.559 W srv llama_server: -----------------
0.00.184.564 W srv llama_server: the following feature(s) are enabled:
0.00.184.565 W srv llama_server: built-in tools (experimental)
0.00.184.565 W srv llama_server: do not expose the server to untrusted environments
0.00.184.566 W srv llama_server: -----------------
0.00.186.919 I srv load_model: loading model './Qwen3.6-27B-UD-Q2_K_XL.gguf'
0.01.045.611 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
1.02.264.707 I cmn init: llama threadpool init, n_threads = 6
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend-meta.cpp:1156: multi buffers are not supported by the meta backend
[New LWP 59421]
[New LWP 59420]
[New LWP 59419]
[New LWP 59418]
[New LWP 59417]
[New LWP 59416]
[New LWP 59415]
[New LWP 59414]
[New LWP 59413]
[New LWP 59412]
[New LWP 59411]
[New LWP 59410]
[New LWP 59409]
[New LWP 59408]
[New LWP 59407]
[New LWP 59406]
[New LWP 59405]
[New LWP 59404]
[New LWP 59403]
[New LWP 59402]
[New LWP 59401]
[New LWP 59400]

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
⚠️ warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56 in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1 0x000076e1862a061c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
⚠️ warning: 49 ./nptl/cancellation.c: No such file or directory
#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75 in ./nptl/cancellation.c
#3 0x000076e18631cd4f in __GI___wait4 (pid=, stat_loc=, options=, usage=) at ../sysdeps/unix/sysv/linux/wait4.c:30
⚠️ warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4 0x000076e18694fcdb in ggml_print_backtrace () from /home/aryan/Desktop/AI/libggml-base.so.0
#5 0x000076e18694fe72 in ggml_abort () from /home/aryan/Desktop/AI/libggml-base.so.0
#6 0x000076e18697e3e4 in ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_meta_simple_tensor_container&, ggml_tensor*) () from /home/aryan/Desktop/AI/libggml-base.so.0
#7 0x000076e186967773 in ggml_gallocr_alloc_graph () from /home/aryan/Desktop/AI/libggml-base.so.0
#8 0x000076e18696de21 in ggml_backend_sched_alloc_graph () from /home/aryan/Desktop/AI/libggml-base.so.0
#9 0x000076e1858f5a7e in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/aryan/Desktop/AI/libllama.so.0
#10 0x000076e1858fce17 in llama_context::decode(llama_batch const&) () from /home/aryan/Desktop/AI/libllama.so.0
#11 0x000076e1858fe1c0 in llama_decode () from /home/aryan/Desktop/AI/libllama.so.0
#12 0x000076e185e45330 in common_init_from_params(common_params&, bool) () from /home/aryan/Desktop/AI/libllama-common.so.0
#13 0x000076e186bc26c3 in server_context_impl::load_model(common_params&) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#14 0x000076e186aef648 in llama_server(common_params&, int, char**) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#15 0x000076e186af0ecf in llama_server(int, char**) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#16 0x000076e18622a601 in __libc_start_call_main (main=main@entry=0x571d7ef1f260

, argc=argc@entry=27, argv=argv@entry=0x7ffdad51c3f8) at ../sysdeps/nptl/libc_start_call_main.h:59
⚠️ warning: 59 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#17 0x000076e18622a718 in __libc_start_main_impl (main=0x571d7ef1f260 , argc=27, argv=0x7ffdad51c3f8, init=, fini=, rtld_fini=, stack_end=0x7ffdad51c3e8) at ../csu/libc-start.c:360
⚠️ warning: 360 ../csu/libc-start.c: No such file or directory
#18 0x0000571d7ef1f295 in _start ()
[Inferior 1 (process 59395) detached]
Aborted (core dumped) systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ../llama-server -m ./Qwen3.6-27B-UD-Q2_K_XL.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b -ctk q8_0 -ctv q8_0 --tools all -fa on -sm tensor --spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8
aryan@hope:~/Desktop/AI/best$