Eval bug: Metal: GGML_ASSERT(buf_src) failed in ggml_metal_buffer_set_tensor on Intel Mac with AMD discrete GPU (newBufferWithBytesNoCopy page-alignment requirement) #26949

Name and Version

Built from source at tag b10358 (shallow clone, so the binary reports build: b1-030ebb5; commit 030ebb5). Also reproduced on b6440 (33daece).

Operating systems

Mac

GGML backends

Metal

Hardware

Intel Xeon W (iMac Pro 2017, hw.model = iMacPro1,1) + AMD Radeon Pro Vega 56 (8 GB VRAM, discrete GPU over PCIe)

Models

Qwen_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf

Problem description & steps to reproduce

Steps to reproduce:

Build at tag b10358 on macOS Big Sur 11.7 with Apple clang 13 (CommandLineTools): cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON && cmake --build build --config Release
Run any GGUF with GPU offload enabled: ./build/bin/llama-cli -m model.gguf -ngl 1 -p "hello"
It aborts with GGML_ASSERT(buf_src) failed at ggml/src/ggml-metal/ggml-metal-device.m:1862

Problem description

On an Intel Mac with a discrete AMD GPU, any run with GPU offload enabled aborts during the first decode with:

GGML_ASSERT(buf_src) failed at ggml/src/ggml-metal/ggml-metal-device.m:1862

This happens with -ngl 1 as well as -ngl 999, so offloading a single tensor is enough to trigger it. The build itself succeeds (Apple clang 13 from CommandLineTools, cmake -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON).

Call path: llama_decode → graph_compute → ggml_backend_sched_graph_compute_async → ggml_metal_buffer_set_tensor.

Root cause

In ggml_metal_buffer_set_tensor, the raw tensor pointer is handed to newBufferWithBytesNoCopy:

void * data_ptr = (void *)(uintptr_t) data;
id buf_src = [buf->dev->mtl_device newBufferWithBytesNoCopy:data_ptr length:size options:MTLResourceStorageModeShared deallocator:nil];
GGML_ASSERT(buf_src);
newBufferWithBytesNoCopy requires the pointer to be page-aligned (4096 bytes on Intel Macs) and returns nil otherwise. Tensor data loaded from a GGUF file is tightly packed, so this alignment is not guaranteed. buf_src becomes nil and the assert fires.

This path is only reached on non-unified-memory devices. The function starts with:

if (buf->is_shared) { memcpy((char *) tensor->data + offset, data, size); return; }
On Apple Silicon is_shared is true and the memcpy path returns early, so the newBufferWithBytesNoCopy path is never exercised there. That is consistent with the README listing Metal as an Apple Silicon backend, but the path is still compiled and taken on Intel + discrete AMD.

The same pattern exists in ggml_metal_buffer_get_tensor (line 1914), which passes the destination pointer directly. The other three uses of newBufferWithBytesNoCopy in this file (lines 1664, 1733, 1760) operate on already-aligned pointers and sizes and are fine.

Not a gpuAddress issue. MTLBuffer.gpuAddress is macOS 13.0+, and its absence does break builds on older SDKs, but that is a separate problem: the same assert is reported on macOS 13.7.8 (Intel iMac 2017 + Radeon Pro 580), where gpuAddress does exist.

GGML_METAL_CONCURRENCY_DISABLE=1 does not help. The variable is implemented (ggml-metal-context.m:137, res->use_concurrency = getenv("GGML_METAL_CONCURRENCY_DISABLE") == nil), but the abort occurs at the identical line with an identical stack.

Suggested fix

Allocate a buffer and copy into it, instead of wrapping an existing pointer. Cost is one memcpy.

set_tensor:

id buf_src = [buf->dev->mtl_device newBufferWithLength:size options:MTLResourceStorageModeShared];
GGML_ASSERT(buf_src);
memcpy([buf_src contents], data, size);
get_tensor:

id buf_dst = [buf->dev->mtl_device newBufferWithLength:size options:MTLResourceStorageModeShared];
GGML_ASSERT(buf_dst);
and, immediately AFTER [cmd_buf waitUntilCompleted]: memcpy(data, [buf_dst contents], size);
The ordering of the get_tensor memcpy matters: placing it before waitUntilCompleted reads the buffer before the blit completes, which does not crash but silently corrupts the data.

With these two changes the model loads and runs. Output is correct (prompt "hello" returns "Hello! How can I assist you today?"); I did not observe the output corruption reported for AMD discrete GPUs in #19563.

Secondary finding 1: most operations are not dispatched to Metal at all

has_simdgroup_reduction is set from two conditions only (ggml-metal-device.m:731-732):

supportsFamily:MTLGPUFamilyApple7
supportsFamily:MTLGPUFamilyMetal3_GGML, where MTLGPUFamilyMetal3_GGML is a hard-coded 5001 (line 26) because the constant is absent from the Big Sur SDK
Neither holds on Vega 56 under Big Sur, so has_simdgroup_reduction is false, and the twelve supports_op cases that return it (including GGML_OP_MUL_MAT) all return false. The scheduler consequently runs those ops on the CPU. The log confirms this directly:

resolve_fused_ops: layer 0 is assigned to device MTL0 but Flash Attention is assigned to device CPU (usually due to missing support)
resolve_fused_ops: Flash Attention not supported, set to disabled
So the backend loads, runs, and produces correct output on pre-Metal3 AMD hardware — it simply does almost no work on the GPU. This is a more precise statement than "the Metal backend does not work on pre-Metal3 AMD GPUs", which is the conclusion currently recorded in #15228.

Secondary finding 2: weights are not resident in VRAM

After the fix the load log reports the weights on the Metal side:

load_tensors: offloaded 37/37 layers to GPU
load_tensors: MTL0_Mapped model buffer size = 4275.33 MiB
llama_kv_cache: MTL0_Private KV buffer size = 576.00 MiB
sched_reserve: MTL0_Private compute buffer size = 32.00 MiB
Monitoring VRAM with ioreg during inference:

idle: 712.2 MiB used
after model load: 1320.5 MiB used
during generation: 1321.4 MiB used, device utilization 10%, core clock 71 MHz, fan 0 RPM
The increase is 608.3 MiB, matching MTL0_Private KV (576.00) + compute (32.00) exactly. Nothing corresponding to the 4275.33 MiB of MTL0_Mapped weights is ever resident: the weights stay in host memory and are read over PCIe. This matches #15228.

Measured throughput on Qwen3 8B Q4_K_M is therefore not improved by offload: 17.4 t/s prompt with -ngl 999 -c 4096, against 53.4 t/s prompt / 12.9 t/s generation for CPU-only. -c 4096 is correctly applied (n_ctx = 4096 and KV 576.00 MiB confirmed in the log), and --no-repack makes it worse (14.7 t/s vs 17.4 t/s).

Diagnosability note

The device property block that logs GPU name, GPU family, simdgroup reduction, simdgroup matrix mul., has unified memory, has bfloat and has tensor (ggml-metal-device.m around line 936) never appears in any output on this configuration. It is missing from llama-cli with -v --log-file, from llama-cli captured through script, and from llama-server with -v, while the ggml_metal_init lines (allocating / found device / picking default device / use fusion / use concurrency / use graph optimize) are printed normally, twice. As a result a user cannot tell from the logs whether their GPU satisfies has_simdgroup_reduction. Making that block reliably visible would save a lot of guesswork on non-Apple-Silicon hardware.

Related issues

#15228 — weights mapped with Shared storage, read over PCIe on discrete GPUs (reopened 2026-02-19)
#19527 — PR addressing Metal performance on AMD discrete GPUs
#19563 — output corruption on AMD discrete GPUs
#25866 — GCN/Vega wave64 specific issue
I am happy to open a PR with the two-line fix if that is useful.

First Bad Commit

No response

Relevant log output