Repository navigation
ggml-backend-meta: multi buffers are unsupported leading to vulkan segfault #22197
Description
Name and Version
version: 8863 (9789512)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
Vulkan
Hardware
Intel(R) Xeon(R) CPU E5-2630L v3
2× AMD Radeon RX 480 (Polaris10, 4 GB each, 8 GB aggregate VRAM)
Linux 6.8.0-110-generic (Ubuntu 24.04)
Mesa 25.2.8 / RADV driver
No peer-to-peer; PCIe 3.0 x16
Models
Llama 3.2 3B (unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M)
Problem description & steps to reproduce
Summary
llama-cli segfaults during model load when using --split-mode tensor on Vulkan unless the user explicitly passes -c with a "small" context size.
Repro
./build/bin/llama-cli \
-hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M \
-ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
Result: Segmentation fault (core dumped) after "Loading model..." spinner.
Workaround
Any small -c avoids the crash, this works fine:
./build/bin/llama-cli ... -c 2048
First Bad Commit
No response
Relevant log output
Logs
$ gdb --args ./build/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
GNU gdb (Ubuntu 15.1-1ubuntu1~24.04.1) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/bin/llama-cli...
(No debugging symbols found in ./build/bin/llama-cli)
(gdb) r
Starting program: /home/matto/llama.cpp/build/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe29ff6c0 (LWP 366687)]
[New Thread 0x7fffe21fe6c0 (LWP 366688)]
[New Thread 0x7fffe18bc6c0 (LWP 366689)]
[New Thread 0x7fffe0f7a6c0 (LWP 366690)]
[New Thread 0x7fffd3fff6c0 (LWP 366692)]
[Thread 0x7fffd3fff6c0 (LWP 366692) exited]
[New Thread 0x7fffd3fff6c0 (LWP 366693)]
[Thread 0x7fffd3fff6c0 (LWP 366693) exited]
[New Thread 0x7fffd3fff6c0 (LWP 366694)]
[Thread 0x7fffd3fff6c0 (LWP 366694) exited]
[New Thread 0x7fffd3fff6c0 (LWP 366695)]
Loading model... |[New Thread 0x7fffd37fe6c0 (LWP 366696)]
/[New Thread 0x7fffd08886c0 (LWP 366743)]
[Thread 0x7fffd08886c0 (LWP 366743) exited]
[New Thread 0x7fffd08886c0 (LWP 366744)]
[Thread 0x7fffd08886c0 (LWP 366744) exited]
[New Thread 0x7fffd08886c0 (LWP 366745)]
[Thread 0x7fffd08886c0 (LWP 366745) exited]
[New Thread 0x7fffd08886c0 (LWP 366746)]
[Thread 0x7fffd08886c0 (LWP 366746) exited]
-
Thread 1 "llama-cli" received signal SIGSEGV, Segmentation fault.
0x00007ffff3970fb0 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/matto/llama.cpp/build/bin/libggml-vulkan.so.0
(gdb) bt full
#0 0x00007ffff3970fb0 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/matto/llama.cpp/build/bin/libggml-vulkan.so.0
No symbol table info available.
#1 0x00007ffff797fcb5 in ggml_backend_meta_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/matto/llama.cpp/build/bin/libggml-base.so.0
No symbol table info available.
#2 0x00007ffff7975877 in ggml_backend_sched_graph_compute_async () from /home/matto/llama.cpp/build/bin/libggml-base.so.0
No symbol table info available.
#3 0x00007ffff76bc461 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#4 0x00007ffff76be574 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#5 0x00007ffff76c5920 in llama_context::decode(llama_batch const&) () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#6 0x00007ffff76c73ef in llama_decode () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#7 0x00007ffff7bfa59f in common_init_from_params(common_params&) () from /home/matto/llama.cpp/build/bin/libllama-common.so.0
No symbol table info available.
#8 0x0000555555673068 in server_context_impl::load_model(common_params&) ()
No symbol table info available.
#9 0x00005555555a082e in main ()
No symbol table info available.Reacted by Evan Rovelli, Ross Rosario and Jesper Fyhr Knudsen
Activity
mattngaw commentedon Apr 21, 2026
gdb backtrace on the debug llama-cli:
Logs
I intend to continue digging into this myself, if anyone could verify reproducibility that'd be great.
LostRuins commentedon May 2, 2026
@mattngaw I can reproduce on vulkan with Qwen3 4B (full offload) and it segfaulted, somewhere in
ggml_vk_tensors_overlap (elementwise=false, b=0x22c1e16b3a0, a=0x22cac9a9ba0)atllama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp
Line 14495 in 63d93d1
Testing with a single GPU, laptop rtx 4090.
#19378 (comment)
TheBlueMatt commentedon May 31, 2026
This may be fixed by TheBlueMatt@6e113cb someone needs to clean it up and upstream at some point...
d-shehu commentedon Jun 1, 2026
Any chance this will eventually work with AMD GPUs using Vulkan or is explicitly Nvidia-centric?
Would it also work with RPC using mix GPUs (RADv Vulkan + Nvidia Cuda)?
d-shehu commentedon Jun 3, 2026
I just tested b9494 and this still crashes with OOM, alloc_tensor_range, with every model (example: Qwen 3.6 27b and 3.5 122b).
"Layer" or "row" work fine.
netrunnereve commentedon Jun 15, 2026
I'm seeing this as well and finally got some time to look into it. Basically what happens is that when the KV cache gets too big the Vulkan backend allocates multiple buffers to fit it in memory rather than a single buffer, for me that triggers around 1GB or so. If you set
GGML_VK_MEMORY_LOGGER=1and mess with the context size then there's a point where the backend will decide to split up the buffers.I think CUDA is able to allocate large buffers in one go so it doesn't have this problem, but I can't test this. Anyways when the backend uses multiple buffers to store all the data a ggml multi_buffer gets created, which wraps multiple buffers together in a single object (the tensors are still assigned to actual buffers). I think it's done that way so that ggml actually knows that there are several buffers and that they need to be deleted together and all that. The multi_buffer itself doesn't have a valid context and we get our segfault.
Here b is a multi buffer. @TheBlueMatt's patch seems to work as it uses the original Vulkan source of the buffer instead of the multi buffer. However it depends on the fact that the tensor has a valid view src (is that always the case?) that links to the original buffer.
If we have a small KV cache and don't have a multi buffer then the meta backend delivers a regular Vulkan buffer (I checked). I'm still looking into it but basically the meta backend was created assuming that there's only one buffer per ggml context.
So this explains why it crashes but don't know the code well enough to actually confirm that his patch fixes this properly.
JohnCMcDonough commentedon Jun 16, 2026
I think I’m hitting the same bug here, once I push
-c/--ctx-sizehigh enough. I'm still figuring out llama.cpp, so I hope this is helpful.Behavior so far:
Hardware:
Bluefin 44 / Fedora 44 base6.19.14-101.fc44.x86_64AMD Ryzen 7 9800X3D60.4 GiBAMD Radeon AI PRO R9700(gfx1201),32 GB VRAMAMD Radeon AI PRO R9700(gfx1201),32 GB VRAMMesa RADVllama-server --version:9660 (7dad2f1a1)I can reproduce it reliably with:
(I'm pretty sure backend sampling doesn't work with tensor split though, so that can safely be removed).
With this setup, the process exits with 139. The kernel log shows a segfault in libggml-vulkan.so after model load, around slot initialization.
Full gdb trace
netrunnereve commentedon Jun 17, 2026
Okay this patch is only going to work if the tensor in the multi buffer is a view tensor. If it's the original source of the data then it's not going to have a view_src and the patched function isn't going to return the correct Vulkan buffer.
I dug into this and how the multi buffer works is that it basically allocates multiple buffers on the backend when you do a
ggml_backend_alloc, and it returns a multi buffer object instead of a backend buffer. The multi buffer in turn contains an array of pointers to the backend buffers and it has some functions allowing you to say free or clear all the associated buffers in one go. In fact multi buffers were actually created specifically by @0cc4m for the Vulkan backend a long time ago.Since with multi buffers it's possible for tensors to be stored in different physical buffers, during the allocation process each tensor has its associated
tensor->bufferset to identify where it's actually stored. The tensors don't know or care if it's in a multi buffer and when we run the model the backend just goes totensor->bufferto get the actual location of the data.Now the meta backend used for tensor parallel is a virtual backend that picks up the graph like any other backend and internally splits it up so that it runs in parallel across different backend devices. It does a
ggml_backend_allocfor each backend and it keeps track of the returned buffer. That buffer is then passed on to the meta backend context tensors under the assumption that each backend has a single buffer for each context. So in our case with a multi buffertensor->bufferends up pointing to a multi buffer and we get our segfault as the Vulkan backend expects a single Vulkan buffer. I thought of modifying the Vulkan backend to make it support multi buffers but that requires it to know which buffer in the multi buffer is storing that specific tensor, and we don't know that.What I think we can do is:
ggml_backend_allocwhen it returns a multi buffer, and later when we init the meta backend transfer those buffers over to the new meta tensors. Obviously we would have to ensure that the meta tensor being initialized is the same as the one that was allocated on the backend in the first place.Of course I might be missing something here and I'm not going to pretend that I understand the meta backend fully as I don't!
ggml-backend-meta: fix SPLIT_MODE_TENSOR segfault on multi_buffer KV-…
[-]Eval bug: segfault under Vulkan --split-mode tensor without explicit -c[/-][+]ggml-backend-meta: multi buffers are unsupported leading to vulkan segfault[/+]aryan-nikzad commentedon Aug 18, 2026
4x rx580
2620v3
ga x99psli
above 4g on (modgrub+setupvar hack)
rebar on (patched by ReBarUEFI)
3.6 35b a3b in q4kxl works @ 35tok/s generation
aryan@hope:~/Desktop/AI/best$ systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ../llama-server -m ./Qwen3.6-27B-UD-Q2_K_XL.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b --tools all -fa on -sm tensor --spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8 -c 8192
Running as unit: run-p62239-i78712.scope; invocation ID: 25d4e80c6c9d4af1a98f7d8b17e54041
0.00.163.372 W server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)
0.00.163.528 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the
-lv NCLI arg)0.00.168.563 W srv llama_server: -----------------
0.00.168.566 W srv llama_server: the following feature(s) are enabled:
0.00.168.567 W srv llama_server: built-in tools (experimental)
0.00.168.567 W srv llama_server: do not expose the server to untrusted environments
0.00.168.567 W srv llama_server: -----------------
0.00.170.749 I srv load_model: loading model './Qwen3.6-27B-UD-Q2_K_XL.gguf'
0.00.992.688 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
0.58.568.792 I cmn init: llama threadpool init, n_threads = 6
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend-meta.cpp:1156: multi buffers are not supported by the meta backend
[New LWP 62265]
[New LWP 62264]
[New LWP 62263]
[New LWP 62262]
[New LWP 62261]
[New LWP 62260]
[New LWP 62259]
[New LWP 62258]
[New LWP 62257]
[New LWP 62256]
[New LWP 62255]
[New LWP 62254]
[New LWP 62253]
[New LWP 62252]
[New LWP 62251]
[New LWP 62250]
[New LWP 62249]
[New LWP 62248]
[New LWP 62247]
[New LWP 62246]
[New LWP 62245]
[New LWP 62244]
This GDB supports auto-downloading debuginfo from the following URLs:
⚠️ warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
⚠️ warning: 49 ./nptl/cancellation.c: No such file or directory
⚠️ warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
, argc=argc@entry=25, argv=argv@entry=0x7ffd5ffba0f8) at ../sysdeps/nptl/libc_start_call_main.h:59https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56 in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1 0x000079707c0a061c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75 in ./nptl/cancellation.c
#3 0x000079707c11cd4f in __GI___wait4 (pid=, stat_loc=, options=, usage=) at ../sysdeps/unix/sysv/linux/wait4.c:30
#4 0x000079707c74fcdb in ggml_print_backtrace () from /home/aryan/Desktop/AI/libggml-base.so.0
#5 0x000079707c74fe72 in ggml_abort () from /home/aryan/Desktop/AI/libggml-base.so.0
#6 0x000079707c77e3e4 in ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_meta_simple_tensor_container&, ggml_tensor*) () from /home/aryan/Desktop/AI/libggml-base.so.0
#7 0x000079707c767773 in ggml_gallocr_alloc_graph () from /home/aryan/Desktop/AI/libggml-base.so.0
#8 0x000079707c76de21 in ggml_backend_sched_alloc_graph () from /home/aryan/Desktop/AI/libggml-base.so.0
#9 0x000079707b6f5a7e in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/aryan/Desktop/AI/libllama.so.0
#10 0x000079707b6fce17 in llama_context::decode(llama_batch const&) () from /home/aryan/Desktop/AI/libllama.so.0
#11 0x000079707b6fe1c0 in llama_decode () from /home/aryan/Desktop/AI/libllama.so.0
#12 0x000079707bc45330 in common_init_from_params(common_params&, bool) () from /home/aryan/Desktop/AI/libllama-common.so.0
#13 0x000079707c9c26c3 in server_context_impl::load_model(common_params&) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#14 0x000079707c8ef648 in llama_server(common_params&, int, char**) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#15 0x000079707c8f0ecf in llama_server(int, char**) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#16 0x000079707c02a601 in __libc_start_call_main (main=main@entry=0x57c410044260
#17 0x000079707c02a718 in __libc_start_main_impl (main=0x57c410044260 , argc=25, argv=0x7ffd5ffba0f8, init=, fini=, rtld_fini=, stack_end=0x7ffd5ffba0e8) at ../csu/libc-start.c:360
#18 0x000057c410044295 in _start ()
[Inferior 1 (process 62239) detached]
Aborted (core dumped) systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ../llama-server -m ./Qwen3.6-27B-UD-Q2_K_XL.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b --tools all -fa on -sm tensor --spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8 -c 8192
0cc4m commentedon Sep 23, 2026
Check again now that #29266 is merged, the crash should no longer happen for many cases. Performance is still expected to be bad compared to layer split.
netrunnereve commentedon Sep 23, 2026
Just tested and it's not crashing for me anymore! Going to close this.