Metal: --embeddings with pooling still builds and computes the LM head, producing a multi-GiB discarded tensor and returning all-NaN embeddings on long inputs #27784

Summary

When llama-server --embeddings runs a pooled embedding model on the Metal backend, the graph still contains MUL_MAT result_output (the LM head). Its output is never read — result_embd_pooled is a GET_ROWS from result_norm — but it is fully materialized on every request.

For Qwen3-Embedding-0.6B (vocab 151669) at 7343 tokens that discarded tensor is f32 [151669, 7343] = 4,454,821,868 bytes (4.15 GiB), and it grows linearly with input length. Past some input size the Metal backend then produces out-of-bounds behavior and the server returns all-NaN vectors. Gating the LM head behind an embeddings + pooling check eliminates the failure and drops the MTL0 compute buffer by roughly 10×.

Symptom

Embeddings come back as all-NaN. nlohmann/json serializes NaN as null, so a client sees a vector of null rather than an error.

• /health continues returning 200
• no Metal error is logged
• no recovery short of restarting the process
• -ngl 0 (CPU) is completely clean

The onset depends on how far past the threshold the input is:

• Near the threshold (~7343 tokens at vocab 151669): the triggering request itself returns a valid vector, and every request after it — including a 4-token input — returns all-NaN for the lifetime of the process. This is consistent with the overrun landing in a mapped neighbor allocation rather than in the triggering request's own output.
• Well past it (28,001 tokens): the triggering request is already all-NaN itself.

Worth stating explicitly because the near-threshold case is the dangerous one — the request that breaks the server looks successful, so a pipeline that validates only the newest response never sees a failure.

Environment

Reproduced on unmodified upstream master:

• build 1 (192067b), AppleClang 21.0.0.21000101, Darwin arm64, shallow clone of ggml-org/llama.cpp, -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
• Apple M5 Max, 128 GB unified memory (110100 MiB visible to Metal), macOS
• llama-server --embeddings, Qwen3-Embedding-0.6B f16, -c 32768 -np 1 -b 8192 -ub 8192 -ngl 99

Originally found on two independent forks; the pristine build above was made specifically to rule out fork-local Metal changes, and reproduces identically.

Reproduction

llama-server -m Qwen3-Embedding-0.6B-f16.gguf --embeddings \
  -ngl 99 -c 32768 -b 8192 -ub 8192 -np 1 --host 127.0.0.1 --port 8084
python3 -c "print('가나다라마바사한글테스트문장입니다'*2000, end='')" > /tmp/long.txt
curl -s localhost:8084/v1/embeddings -H 'Content-Type: application/json' </span>

-d "$(python3 -c "import json;print(json.dumps({'input':open('/tmp/long.txt').read(),'model':'emb'}))")" </span>

| python3 -c "import sys,json;d=json.load(sys.stdin);v=d['data'][0]['embedding'];print('prompt_tokens',d['usage']['prompt_tokens'],'nan',sum(1 for x in v if x is None or x!=x),'/',len(v))"
curl -s localhost:8084/v1/embeddings -H 'Content-Type: application/json' </span>

-d '{"input":"hello","model":"emb"}' </span>

| python3 -c "import sys,json;v=json.load(sys.stdin)['data'][0]['embedding'];print('nan',sum(1 for x in v if x is None or x!=x),'/',len(v))"

Observed on the pristine build:

prompt_tokens 28001 nan 1024 / 1024
nan 1024 / 1024

Any non-Latin text works; the point is only to exceed the token threshold.

Adding --no-kv-offload converts the silent corruption into a deterministic hard fault:

ggml_metal_synchronize: command buffer 1 failed with status 5
error: Caused GPU Address Fault Error (kIOGPUCommandBufferCallbackErrorPageFault)

which is what made the offending node identifiable. The reason it does not normally fault is that the overrun lands inside a mapped neighbor allocation — the KV cache. Pull KV off the GPU and that region is unmapped, so the same access page-faults.

Locating the node

With -nkvo for determinism and GGML_METAL_GRAPH_DEBUG=2 for per-node logging (note: this requires -lv 5 — LOG_LEVEL_DEBUG is 5 in common/log.h and common/log.cpp drops all DEBUG output when verbosity < 5, so -lv 4 silently logs nothing):

node[  14] - MUL_MAT      result_output
  src0 = [ 1024, 151669, 1, 1] f16
  src1 = [ 1024,   7343, 1, 1] f32
  node = [151669,  7343, 1, 1] f32   nb = [4, 606676, 4454821868, ...]
node[  15] - GET_ROWS     result_embd_pooled
  src0 = result_norm
E ggml_metal_synchronize: command buffer 1 failed with status 5

Node 15 confirms the pooled embedding comes from result_norm, so node 14's 4.15 GiB output is dead weight.

Cause

src/models/qwen3.cpp appends the LM head with no check on cparams.embeddings:

cur = build_lora_mm(model.output, cur, model.output_s);
cb(cur, "result_output", -1);
res->t_logits = cur;
ggml_build_forward_expand(gf, cur);

set_outputs in src/llama-graph.cpp likewise marks t_logits as a graph output unconditionally.

Proposed fix (local, per-architecture)

// logits are unused in embeddings mode with pooling
if (!cparams.embeddings || pooling_type == LLAMA_POOLING_TYPE_NONE) {
    cur = build_lora_mm(model.output, cur, model.output_s);
    cb(cur, "result_output", -1);
    res->t_logits = cur;
    ggml_build_forward_expand(gf, cur);
}

Both cparams and pooling_type are already members of llm_graph_context, so no new includes are needed.

I applied this only to qwen3.cpp. The other architecture files build the LM head unconditionally as well, so this is not Qwen3-specific — any large-vocab model with a long enough context should reproduce it. A general fix at the llama-graph.cpp level is probably preferable to patching each architecture.

Results

Pristine upstream master, same config as above:

Note also that MTL_SHADER_VALIDATION=1 reports nothing here. ggml-metal binds one large backend buffer and addresses tensors by offset, so writing into a neighbor tensor stays within the bound buffer's length and is legal from the validator's point of view.

Checking whether a build is affected

Look for MTL0 compute buffer size in the startup log — note this line only appears at -lv 5, the same gating that hides the ggml node logs, so if your launcher does not pass -lv it will not be in your log at all. For Qwen3-Embedding-0.6B at -c 32768 -ub 8192, ~5300 MiB means the LM head is being built; ~475 MiB means it is not.

Separately, and regardless of this issue: an embedding pipeline should validate x is None or x != x per component. A length-only check passes happily when all 1024 components are NaN.