Eval bug: Vulkan/RADV Radeon 680M regression: --load-mode auto causes device loss/OOM on large Gemma model; --load-mode mmap fixes it #27360

Activity

1. dallowayk

   added

   bug-unconfirmed

   on Aug 19, 2026
2. tobiasvandriessel commented on Sep 2, 2026

   

   tobiasvandriessel

   on Sep 2, 2026

   More actions

   I’m also impacted by this problem and the mmap mode indeed resolves the issue!

   👍React with 👍1Reacted by dallowayk
3. foolish-dev commented on Sep 11, 2026

   

   foolish-dev

   on Sep 11, 2026

   More actions

   Same failure here on a Strix Halo APU (Ryzen AI Max+ 395 / Radeon 8060S, gfx1151): 4 GiB carve-out, 13.52 GiB GTT, 27.0 GiB RAM visible to Linux, the same layout as the 680M report above.

   Setup: RADV (Mesa 26.2.2), Linux 7.2.3-1-cachyos, llama-server built from a2878d3 (build 10909) with -DGGML_VULKAN=ON, dense 27B Q4_K_M (qwen35 arch, ~16 GiB), --jinja -c 16384, default --load-mode auto. All 3 runs (1 with default --fit, 2 with -ngl 32) aborted in the warm-up decode with vk::DeviceLostError: vk::Queue::submit: ErrorDeviceLost.

   The kernel log for those runs has 1560x amdgpu 0000:c4:00.0: [drm] *ERROR* amdgpu_vm_validate() failed. and 1563x amdgpu 0000:c4:00.0: [drm] *ERROR* Not enough memory for command submission!. There are no ring timeout or ring reset lines in that boot.

   Likely mechanism

   Line numbers are at a2878d3; no commit up to master 8172e657 touches these files. The code facts are cited; that they cause the crash is my inference.

   1. iGPU -> mmap off. Vulkan reports no mmap support on iGPUs (ggml/src/ggml-vulkan/ggml-vulkan.cpp:19116, from llama: add default load-mode auto, which avoids mmap on iGPUs #26081), so --load-mode auto, the default (common/common.h:484, src/llama-model.cpp:2766), turns mmap off (src/llama-model.cpp:1417-1426).
   2. mmap off -> weights in Vulkan_Host. Non-offloaded layers and the input layer use the CPU weight list (src/llama-model.cpp:1498-1500, :1510): ACCEL -> GPU host -> CPU extra -> CPU, with the host type added unless no_host is set (:1033-1063, :1055). The loader takes the first entry whose device supports the weight's op (src/llama-model-loader.cpp:1060-1071) and swaps a host buffer type back to CPU only when mmap is on (:1262-1270), which is why --load-mode mmap works around this. So the CPU-side weights that Vulkan can run stay in Vulkan_Host instead of CPU/CPU_REPACK. This is the same path as Eval bug: llama_model_load: error loading model: vk::Queue::submit: ErrorDeviceLost #18962; as I read it, llama: add default load-mode auto, which avoids mmap on iGPUs #26081 made it the default on iGPUs.
   3. On a RADV APU, Vulkan_Host is GTT, and so is anything Vulkan0 can't fit in the carve-out. Vulkan_Host is allocated by ggml_vk_host_malloc (ggml-vulkan.cpp:16897-16914), which first asks for HOST_VISIBLE|HOST_COHERENT|HOST_CACHED (:8204-8208); in Mesa 26.2.2 those memory types are RADEON_DOMAIN_GTT (src/amd/vulkan/radv_physical_device.c:571-586). Offloaded weights on UMA prefer DEVICE_LOCAL|HOST_VISIBLE (ggml-vulkan.cpp:3799-3804), and RADV also allows GTT as a placement for VRAM-domain buffers (src/amd/vulkan/winsys/amdgpu/radv_amdgpu_bo.c:565-585), so Vulkan0 memory beyond the 4 GiB carve-out lands in GTT as well. So the whole model sits in carve-out + GTT whatever -ngl is; -ngl only moves weights between Vulkan0 and Vulkan_Host.

   --fit can't see this: its dry run loads with LLAMA_LOAD_MODE_NONE (common/fit.cpp:57), so it places weights in Vulkan_Host too, and it counts every buffer type whose is_host is true as host memory (:77-83). Vulkan_Host reuses the CPU is_host (ggml-vulkan.cpp:16941), and the iGPU's free memory includes the GTT budget (ggml-vulkan.cpp:18987-19009). So GTT is offered to fit as free device memory, while the Vulkan_Host weights that use the same GTT are charged to system RAM.

   Workaround: --no-host (observed)

   These numbers come from llama-server b10760 (0f3a71be), the build bundled in Ollama 0.33.3 (CachyOS ollama 0.33.3-1.1), with libggml-vulkan.so from the matching signed ollama-vulkan 0.33.3-1.1 package, run as a private ollama serve with OLLAMA_IGPU_ENABLE=1 and LLAMA_ARG_NO_HOST=1 (Ollama passes its environment to llama-server). The buffer-placement and fit code is unchanged between b10760 and a2878d3. The 27B is the same GGUF the crash runs used, restamped since: identical tensors, only tokenizer.chat_template differs.

   With --no-host, non-offloaded weights go to CPU and CPU_REPACK buffers. 27B Q4_K_M (17.3 GB file), ctx 4096, q8_0 KV, flash attention on, Ollama num_batch 512, num_gpu = the -ngl column:

   -ngl	Vulkan0 model buffer	CPU + CPU_REPACK model buffers	GPU pool peak (VRAM+GTT, sysfs)	tg tok/s
   4	1729 MiB	~14.8 GiB	3.2 GiB	4.76
   8	2720 MiB	~13.8 GiB	4.2 GiB	4.93
   16	4573 MiB	~11.9 GiB	6.1 GiB	5.31
   24	6356 MiB	~10.1 GiB	7.9 GiB	5.72

   GPU pool usage now tracks the Vulkan0 buffer plus ~1.5-1.7 GiB (KV cache, compute buffers, other GPU clients); the CPU-side weights no longer show up in it. CPU-only generation is ~5.0 tok/s. At -ngl 32 the host ran short of RAM, so I stopped the run. I have no prompt-processing numbers.

   Suggested fix

   When auto turns mmap off because an iGPU reports no mmap support, don't put weights in the device host buffer type (behave as if no_host were set for the weight lists). This puts those weights back on the CPU side, as before #26081 (in CPU/CPU_REPACK buffers, as with --no-host), and keeps #26081's load-time RAM saving. Tradeoff: on UMA the backend can use host buffers directly (ggml-vulkan.cpp:9467-9471), which may help PP when weights are CPU-side; I haven't measured that.

   On the --fit side, common/fit.cpp could charge a host buffer type to its device when that device is GGML_BACKEND_DEVICE_TYPE_IGPU (Vulkan reports that type, ggml-vulkan.cpp:19100; Vulkan_Host's .device is already Vulkan0, :16943). On its own that can't make lowering -ngl help, so fit would have to shrink the context or fail loudly, which is still better than DeviceLost.

   Related: #26081 (introduced auto); #18962 (the same CS ENOMEM via Vulkan_Host with mmap off, fixed by --mmap; closed as stale, it includes an APU report after #26081); #22592 (UMA accounting in --fit).

   👍React with 👍1Reacted by dallowayk
4. github-actions

   added

   stale

   on Oct 12, 2026
5. YuWei-CH commented on Oct 26, 2026

   

   YuWei-CH

   on Oct 26, 2026

   Last edited by YuWei-CH

   More actions

   I can reproduce this issue.

   System:
   APU: AMD Ryzen 7 7735HS / Radeon 680M (RADV REMBRANDT)
   RAM: 32 GB DDR5-4800, 3 GiB VRAM carve-out
   GTT limit: 14,403 MiB (mem_info_gtt_total)
   OS: Ubuntu 24.04.5, kernel 7.0.0-38-generic
   Mesa: 25.2.8
   llama.cpp: d854d98 (b11889), Release build with GGML_VULKAN=ON

   Here are my actions and results:
   I am using Qwen3.6-35B-A3B-Q4_K_M (19.01 GiB) with this command:
   ./llama-server -m Qwen3.6-35B-A3B-Q4_K_M.gguf -v

   With no additional flags, --fit selects all 41 layers for GPU offloading (10 overflowing) and reports:
   Vulkan0: 16140 MiB used, 1034 MiB free
   common_fit_params: successfully fit params to free device memory

   However, the actual allocations include:
   Vulkan0 model buffer size = 15589.26 MiB
   Vulkan_Host model buffer size = 3873.88 MiB
   radv/amdgpu: Not enough memory for command submission.
   ggml_vulkan: device lost on Vulkan0
   llama_model_load: error loading model: vk::Queue::submit: ErrorDeviceLost

   On this CPU, Vulkan0 allocations beyond the 3 GiB carve-out and Vulkan_Host allocations both consume GTT. Using the reported model-buffer sizes, the estimated GTT requirement for weights alone is:
   15589 - 3072 + 3874 = 16391 MiB

   This exceeds the 14,403 MiB GTT limit, which confirm the issue described above: --fit treats Vulkan_Host allocations as host memory, although they consume the same GTT budget as the portion of Vulkan0 memory beyond the carve-out.

   Also, same as above, --no-host allows the same model to run:
   With the same model and default --fit, adding --no-host succeeds:
   Vulkan0 model buffer size = 15729.13 MiB
   CPU model buffer size = 278.01 MiB
   CPU_REPACK model buffer size = 3456.00 MiB
6. github-actions

   removed

   stale

   on Oct 27, 2026