[bogdanr]

Overview

When a Vulkan device is lost mid-graph, vulkan.hpp throws vk::DeviceLostError out of ggml_backend_vk_graph_compute. Nothing catches it, so it unwinds through llama_decode, which is extern "C", and the whole process aborts:

terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Queue::submit: ErrorDeviceLost

The caller cannot defend against this. There is no status to inspect and nothing to catch - the application is gone.

This catches the error and returns GGML_STATUS_FAILED instead, so a lost device fails the call rather than the process.

A lost device invalidates every object created from it, so the flag is sticky and lives on vk_device, not on the backend context: several contexts can share one device and none of them are usable afterwards. Once it is set, every later graph_compute returns GGML_STATUS_FAILED immediately until the backend is recreated.

Two more places needed the same treatment for that to hold:

• ggml_backend_vk_synchronize returns void and can see the same loss, so it records the flag and leaves the reporting to the next graph_compute.
• ggml_vk_cleanup runs from the backend destructor, where a throw aborts just as hard, so its ggml_vk_synchronize call is wrapped too.

No interface change, no other backend touched.

This is the pattern Metal already uses - has_error in ggml-metal-context.m, with the same comment about the backend having to be recreated. This brings Vulkan in line with it.

Additional information

Where I hit it: Intel Lunar Lake integrated graphics (Intel(R) Graphics (LNL)), Mesa Vulkan, xe kernel driver. One HTTP request is enough: prefill ~10,200 tokens of a 26B model with all layers resident. Kernel side, one line per abort:

xe 0000:00:02.0: [drm] Tile0: GT0: Timedout job: seqno=308, ... in fono [3416904]

The trigger is the driver's job watchdog, not anything ggml does wrong. Raising job_timeout_ms from 5000 to 10000 and changing nothing else let the identical prompt finish (471.6 s, no abort). Anything that overruns the per-job deadline will do it, so this is not specific to my hardware - any driver with a watchdog can lose a device this way.

It cannot be avoided from the caller side. Each of these was tried against the same reproducer:

Tried	Result
op_offload off, all layers resident	still aborts
n_ubatch 512 -> 128	still aborts
GGML_VK_MAX_NODES_PER_SUBMIT 100 -> 25	still aborts
GGML_VK_DISABLE_COOPMAT=1, GGML_VK_DISABLE_COOPMAT2=1	still aborts
n_batch 2048 -> 256, three runs	one completed, two aborted

The last row is the telling one: the same setting both survives and dies, so no parameter reliably keeps a job under the deadline.

To be clear about what this does and does not do: it does not stop the device being lost. It stops the device loss taking the process with it. Recovery still means tearing the backend down and building a new one.

What the apps do after the device is lost. The failure comes back as GGML_STATUS_FAILED -> decode returns -3, which both frontends already handle:

• llama-completion / llama-cli: common_prompt_batch_decode logs failed to eval once and returns false, and main returns 1. Clean exit, one message.
• llama-server: ret < -1 maps to Compute error., so every processing slot gets an error response and is released, and update_slots breaks out of the batch loop. The server stays up; a later request fails the same way, one log line each. It does not spin - the retry-with-half-batch path only runs for the KV-space case (ret == 1), never for a compute error.

The sticky flag matters here: once set, graph_compute returns before touching the device at all, so nothing is submitted to a dead queue in a loop. Metal logs on that path per call too, and I kept it the same.

Related: #25664, #27076.

Two things I deliberately left out, happy to add either if you want them here rather than in a follow-up:

• Failing ggml_backend_vk_buffer_* fast once the device is lost. Bigger diff, and the flag is already in the right place to do it later.
• Wrapping llama_encode / llama_decode in try/catch so no backend can unwind across the C boundary. That one is a separate concern from the Vulkan bug; I will open it after this lands.

Builds clean with GGML_VULKAN=ON. I will post the result of re-running the reproducer against a patched build - expected outcome is a failed decode and a live process instead of an abort.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. I hit this crash on my own machine, did the debugging above myself, and decided on the fix. A coding assistant (Claude Code) helped me write the patch and this description under my direction. I have reviewed every line and I am responsible for all of it. The commit carries an Assisted-by: trailer.

[ggml-gh-bot]

Hi @bogdanr, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[bogdanr]

Updated: description now follows the template, and I closed the second PR (#27184) so this one stands alone. Force-pushed the branch with tidier comments and a proper commit message.

[jeffbolznv]

This seems ok. What's the app-level behavior after the device is lost? Will llama-cli continue trying to execute and spam a bunch of error message, or will it cleanly stop?

[bogdanr]

It doesn't continue trying. It produces an error and then releases the processing slot. I've updated the PR message with a better description.

[0cc4m]

Get rid of most of the comments. AI tends to overcomment a lot. Only keep concise hand-written comments where code is unclear.

[0cc4m]

Log an error here.

[0cc4m]

Here too.

[0cc4m]

This text is unhelpful (AI-generated, I guess) and this log is not needed. It should log once when the device is lost, then stop.

[0cc4m]

More AI-generated logging, keep it to the point and avoid suggestions.

[0cc4m]

Also set this to ready for review if it is, it was automatically set to draft because you hadn't used the PR template.