Repository navigation
Eval bug: vk::DeviceLostError on linux-7.x kernels (RADV_STRIXHALO) #25664
Description
Name and Version
$ llama-cli --version
version: 10005 (7f575c3)
built with GNU 15.3.0 for Linux x86_64
Operating systems
Linux 7.0.13
GGML backends
Vulkan
Hardware
Strix Halo (Framework Desktop) 128GB
Models
- unsloth/DeepSeek-V4-Flash-GGUF:UD-IQ3_XXS
- unsloth/Qwen-3.8-27B-GGUF:Q8_0
Problem description & steps to reproduce
users on linux-6.x kernels don't seem to have this issue
ROOT CAUSE: seems to be vulkan op timeouts (various)
CONFOUNDING FACTOR: compute lockup_timeout default for AMD driver changed from 60s to 2s.
WORKAROUND: restore linux 6.x behavior with cmdline: amdgpu.lockup_timeout=10000,60000,10000,10000
model loads fine, but within 2-3 turns, the server crashes with vk::DeviceLostError
the same installation works fine with other models, such as qwen3.6-27b and gemma-4-31b
First Bad Commit
No response
Relevant log output
Logs
3.38.895.141 D srv decode: n_batch (effective) = 2048, off = 0
3.38.895.146 D set_adapters_lora: adapters = (nil)
3.38.895.146 D adapters_lora_are_same: adapters = (nil)
3.38.895.148 D set_embeddings: value = 0
radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent.
3.45.098.561 E srv update_slots: decode() failed: vk::Queue::submit: ErrorDeviceLost
3.45.098.575 E srv send_error: task id = 434, error: decode() failed: vk::Queue::submit: ErrorDeviceLost
3.45.098.577 D res send: sending result for task id = 434
3.45.098.578 D res send: task id = 434 pushed to result queue
3.45.098.582 I slot release: id 0 | task 434 | stop processing: n_tokens = 8713, truncated = 0
3.45.098.584 D slot reset: id 0 | task 434 |
3.45.098.594 D res remove_waiti: remove task 434 from waiting list. current waiting = 1 (before remove)
3.45.098.617 W srv stop: cancel task, id_task = 434
3.45.098.620 D que start_loop: waiting for new tasks
3.45.098.620 D que start_loop: processing new tasks
3.45.098.621 D res remove_waiti: remove task 434 from waiting list. current waiting = 0 (before remove)
3.45.098.623 D que start_loop: processing task, id = 436
3.45.098.626 D que post: new task, id = 437/1, front = 1
3.45.098.630 D srv next: received error result, stopping further processing
3.45.098.632 D que start_loop: processing task, id = 437
3.45.098.633 D que start_loop: update slots
3.45.098.633 I srv update_slots: all slots are idle
3.45.098.634 D que start_loop: waiting for new tasks
3.45.098.667 D srv operator(): error received during streaming, terminating stream
3.45.098.730 D srv operator(): http: streamed chunk: data: {"error":{"code":500,"message":"decode() failed: vk::Queue::submit: ErrorDeviceLost","type":"server_error"}}
3.45.098.739 D srv operator(): http: stream ended
3.45.098.747 D res remove_waiti: remove task 434 from waiting list. current waiting = 0 (before remove)
3.45.098.747 D srv stop: all tasks already finished, no need to cancel
5.19.967.835 D que start_loop: processing new tasks
5.19.967.844 D que start_loop: terminate
5.19.967.846 I srv operator(): operator(): cleaning up before exit...
5.20.017.685 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
5.20.017.688 I common_memory_breakdown_print: | - Vulkan0 (8060S Graphics (RADV STRIX_HALO)) | 127488 = 11059 + (100342 = 97808 + 1763 + 769) + 16086 |
5.20.017.689 I common_memory_breakdown_print: | - Host | 17033 = 414 + 0 + 16618 |
5.20.018.269 W ~llama_context: Vulkan0 compute buffer size of 769.5923 MiB, does not match expectation of 699.3354 MiB
5.20.018.270 D ~llama_context: Vulkan_Host compute buffer size is 16618.8985 MiB, matches expectation of 16618.8985 MiB
/home/user/src/llama.cpp/build/bin/libggml-base.so.0(+0x18565) [0x7fc246f55565]
/home/user/src/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1eb) [0x7fc246f5593b]
/home/user/src/llama.cpp/build/bin/libggml-base.so.0(+0x2b779) [0x7fc246f68779]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbcafa) [0x7fc2472bcafa]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_call_terminate+0x2e) [0x7fc2472a9779]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x9d) [0x7fc2472bc03d]
/usr/lib/x86_64-linux-gnu/libgcc_s.so.1(+0x22463) [0x7fc247c54463]
/usr/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x12b) [0x7fc247c54fab]
/home/user/src/llama.cpp/build/bin/libggml-vulkan.so.0(+0x73093) [0x7fc242a73093]
/home/user/src/llama.cpp/build/bin/libggml-vulkan.so.0(+0xe4837) [0x7fc242ae4837]
/home/user/src/llama.cpp/build/bin/libllama.so.0(_ZN13llama_contextD1Ev+0x731) [0x7fc2466dadb1]
/home/user/src/llama.cpp/build/bin/libllama.so.0(llama_free+0x13) [0x7fc2466db443]
/home/user/src/llama.cpp/build/bin/libllama-common.so.0(_ZN18common_init_resultD1Ev+0xbc) [0x7fc246bf2f2c]
/home/user/src/llama.cpp/build/bin/libllama-server-impl.so(_ZN14server_contextD1Ev+0xa79) [0x7fc24776c739]
/home/user/src/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0xc8a) [0x7fc2476df3ba]
/home/user/src/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0xf26) [0x7fc2476e3b06]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29f77) [0x7fc247033f77]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x87) [0x7fc247034027]
llama-server(+0x11d1) [0x564a8692b1d1]
terminate called after throwing an instance of 'vk::DeviceLostError'
what(): vk::Queue::submit: ErrorDeviceLostReacted by aembr, John Mitrich, badisa, Bobettemcnugget and Anes Belfodil
Activity
[-]Eval bug: vk::DeviceLostError within a few turns on DeepSeekv4-Flash[/-][+]Eval bug: vk::DeviceLostError within a few turns on DeepSeekv4-Flash (RADV_STRIXHALO)[/+]fairydreaming commentedon Jul 14, 2026
Can you add an exact server command line?
khimaros commentedon Jul 14, 2026
@fairydreaming yes, sorry i missed that
khimaros commentedon Jul 14, 2026
for reference, here's the output of llama-fit-params too:
fairydreaming commentedon Jul 15, 2026
@khimaros I tried to reproduce this on my NVIDIA card but no luck. Maybe if you make a debug build with Vulkan debug options
-DGGML_VULKAN=1 -DGGML_VULKAN_DEBUG=1 -DCMAKE_BUILD_TYPE=Debugit will say something more. There's alsoGGML_VULKAN_MEMORY_DEBUGoption that may come handy as well.It may be also worth to see if there are any errors messages in the kernel logs (
sudo dmesg|tail -n 100) after the crash, maybe it's something with the drivers.0cc4m commentedon Jul 15, 2026
Probably a driver timeout in RADV, it won't show up on Nvidia. We have to figure out which operators are timing out.
aembur commentedon Jul 16, 2026
Same here. Hopefully there's enough info in these:
amdgpu_coredump.txt
khimaros commentedon Jul 28, 2026
@aembur which kernel version are you using? i have a sense that i may have started to see more unreliability in the 7.1.x series but i'm not sure yet.
khimaros commentedon Jul 28, 2026
@0cc4m -- i have similar dmesg output to @aembur
i captured a core dump as well, please let me know if it would be helpful to have
khimaros commentedon Jul 28, 2026
@fairydreaming here's the llama-server output with the debug flags enabled:
llama-server.txt
aembur commentedon Jul 28, 2026
@khimaros 7.1.3-arch1-3. This issue did show up pretty recently, and I haven't encountered anything like this before, so I'm inclined to believe your take.
amstan commentedon Jul 28, 2026
I can reproduce it on both 7.1.5-arch1-1 and 7.0.11-arch1-1 with everything else (like mesa userspace and llama.cpp release) being latest.
I even tried downgrading to linux-firmware-amdgpu-20260410 but no luck.
Bobettemcnugget commentedon Aug 1, 2026
I am running to the same issue on Strix Halo, with the new DS4 Flash 0731 images on Lemonade using latest llamacpp build
This is the build currently loaded by Lemonade: llamacpp.vulkan_bin b10218
Thats on Fedora 44 Server 7.1.3-200.fc44.x86_64
Heres dmesg output during two distinct runs, with two distinct llama builds:
asprin13 commentedon Aug 1, 2026
Same problem here. Increasing the timeout in the driver to 3 min worked as a workaround. What I'm observing here is that the GPU is running with quite low load (fan almost idle) and prompt processing takes minutes per batch:
Another thing I noticed are notes in the startup logs about some operations being offloaded to CPU:
khimaros commentedon Aug 5, 2026
i believe i have a root cause identified and a workaround:
Vulkan has no GGML_OP_LIGHTNING_INDEXER implemented so resolve_fused_ops disables fused_lid. DS4 takes fallback at src/models/deepseek4.cpp:600 resulting in a dim0/dim2 swap with consecutive elements 4MB apart.
ggml_vk_get_cpy_pipeline gates tiled transpose on src->nb[1] == ggml_type_size(to) which covers dim0/dim1 only. the dim0/dim2 swap falls through to the generic strided copy. 67m reads with fresh cache line and page.
i have a targeted workaround (<20 LoC adding just tiled transpose for dim0/dim2 swap) along with a complementary backend-ops perf test which triggers the DeviceLost failure without the fix.
implementing GGML_OP_LIGHTNING_INDEXER for Vulkan would probably improve performance on Vulkan while also turning the workaround fix into a backstop or even obviating the need for it entirely.
@fairydreaming what are your thoughts? i can open a PR but want to validate the approach before going any deeper on the op fix.
khimaros commentedon Aug 5, 2026
here's the workaround which has allowed me to run through several rounds of compaction with 256k context:
fairydreaming commentedon Aug 5, 2026
@khimaros Looks like this is already covered by #26585
amstan commentedon Aug 6, 2026
My setup still has a DeviceLostError (using Laguna as the model, but i've seen an identical crash with gemma and qwen3.6 27B).
I tried both the patch above and #26585, same thing.
wowo101 commentedon Aug 6, 2026
Second reproduction here, on a different checkpoint and a much newer build — plus a ROCm control on
the same machine that seems to isolate this to the RADV path.
Setup
gfx1151, 128 GB unifiedamd-strix-halo-toolboxes:vulkan-radvimageunsloth/DeepSeek-V4-Flash-0731-GGUFUD-IQ3_XXS — the 0731 checkpoint, i.e. adifferent model file from the one in your report, at the same quant tier
Identical signature, mid-prefill at
n_tokens16384 of a ~30k-token prompt:It is intermittent. A standalone 30k-token prefill probe completed cleanly; the same operation
under a sustained batch crashed shortly after. Worth stating explicitly because it cost us a wasted
run — one passing probe does not establish stability here.
A possible precursor, if anyone wants a cheaper signal than the crash itself. In the run that
died, prefill throughput degraded within the run — 118 → 59 → 49 tok/s — before the device was
lost. The run average (~95 tok/s) concealed it entirely. If others see the same ramp-down ahead of a
device loss, that is observable long before the queue submit fails.
ROCm control — same box, same model file, same prompt. This is the part that looks most
diagnostic given the
bug-unconfirmedlabel. ROCm 7.14 (b10227) ran 3/3 consecutive cache-busted30k prefills where RADV had died on its second: 307/308/309 s at 94.4/93.9/93.7 tok/s, under 1 %
variance. It also loads 4.4 GiB leaner (98.1 vs 102.5 GiB resident). So with hardware, model file
and workload held constant, the ROCm path is stable where the RADV path is not — which points away
from the model file and away from the hardware.
The machine was undamaged each time: the resident model restarted normally, GTT returned to its
baseline, no lingering device loss.
Caveat: we moved production to ROCm at that point and have not retested RADV since b10224, so
I cannot say whether anything after that changed it. Happy to run a specific RADV repro on request if
it would help confirm the label.
khimaros commentedon Aug 7, 2026
this comment is slop, and the referenced issue above is also not relevant
khimaros commentedon Aug 15, 2026
@fairydreaming i think i found the smoking gun and also why this is not impacting many people (yet).
the 6.x linux kernel previously had a
lockup_timeoutof 60s for compute which dropped to 2s in the 7.x kernel.i'm also running into this (for a different op) with qwen3.8-27b
workaround: restore linux 6.x behavior with cmdline:
amdgpu.lockup_timeout=10000,60000,10000,10000also, @crxtasy12 these massive slop posts are not helpful.
[-]Eval bug: vk::DeviceLostError within a few turns on DeepSeekv4-Flash (RADV_STRIXHALO)[/-][+]Eval bug: vk::DeviceLostError on linux-7.x kernels (RADV_STRIXHALO)[/+]fairydreaming commentedon Aug 15, 2026
@khimaros that sounds similar to what @kuptom found in #21724
But congratulations on getting to the root of the problem!
khimaros commentedon Aug 17, 2026
@fairydreaming i've hit this even with nodes per submit set to 1. some operations just take longer than 2 seconds due to kernel implementation issues in vulkan. for example, the DS4 tiled transpose issue linked above.
AlphasCodes commentedon Aug 17, 2026
I can confirm this issue on my Strix Halo system with the current llama.cpp Vulkan version at around 50% context usage of the configured 262144 context. The amdgpu timeout workaround doesn't help.
I did a test with the llama.cpp ROCm version (via the official Docker version) and I don't run into this issue now. I was able to grow the context usage up to 65% without any issue.
A nice side effect of using the latest llama.cpp ROCm version via official llama.ccp Docker image (ghcr.io/ggml-org/llama.cpp:server-rocm) from 8/16/2026 was way higher prompt processing / prefill numbers (about >= 2x higher) with the same or slightly better token generation numbers even in deeper context (around 100k - 140k).
huppiflupp commentedon Sep 18, 2026
No reproduction on a newer stack: GMKtec EVO-X2 (395, 128 GB), kernel 7.2.0, Mesa 26.2.1,
amdgpu.lockup_timeoutunset (kernel default). llama-server Vulkan, Qwen3.8-27B Q4_K_M,-c 131072 -fa 1, MTP, three chat turns at 38k → 75k → 113k tokens: noErrorDeviceLost, no ring timeout or GPU reset in the kernel log. Also 4 days of daily use up to ~50k context without incident.