Name and Version

broken

version: 9426 (1738129)

last known good

version: 9246 (871b0b7)

Both run via the official ghcr.io/ggml-org/llama.cpp:server-cuda Docker image (-bNNNN tags) on the same host; the version strings above are the build_info reported on /props.

Operating systems

Linux

GGML backends

CUDA

Hardware

Intel Core Ultra 5 245K + 1x NVIDIA GeForce RTX 5070 Ti (16 GB, SM_120 / Blackwell)

Models

Qwen3.6-35B-A3B (MoE), unsloth UD Q2_K_XL GGUF that ships an MTP draft head — file Qwen3.6-35B-A3B-UD-Q2_K_XL-MTP.gguf, plus mmproj-F16.gguf. Run with the built-in MTP draft head via --spec-type draft-mtp.

https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/tree/main

Problem description & steps to reproduce

On b9426, llama-server loads the model, passes /health, and prints server is listening — but the first real completion request crashes the process. The crash happens when the MTP draft path runs its first decode and tries to reserve the prompt-processing compute buffer (~983 MiB): cudaMalloc failed: out of memory → failed to allocate compute pp buffers → std::terminate (backtrace below, through common_speculative_impl_draft_mtp::process → llama_decode).

The identical command and model on b9246 serves this exact workload fine — same -c 131072 --parallel 2, same MTP draft config — so this is a regression, not merely a tight VRAM budget.

Command (b9426 = crashes, b9246 = works):

llama-server -m Qwen3.6-35B-A3B-UD-Q2_K_XL-MTP.gguf \
  --alias Qwen3.6-35B-A3B -ngl 999 -c 131072 --parallel 2 --kv-unified \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
  -fa on --mmproj mmproj-F16.gguf --jinja \
  --chat-template-kwargs '{"enable_thinking":false}' --metrics

Then send a single chat completion. (-ub left at default 512.) Because we run under a container restart policy, the process crash-loops — boot → /health 200 → first request → OOM → restart — which masquerades as a "running/healthy" server that silently fails every request. /health returning 200 is not sufficient: the compute buffers are reserved lazily on the first decode, after /health already reports ready.

Narrowed down: it is per-slot compute-buffer footprint (--parallel bisect)

I isolated the cause by running the identical b9426 build, model, and flags with only --parallel 1 instead of --parallel 2:

• --parallel 1: model loads, first decode returns a real completion (HTTP 200), a longer request decodes 72 tokens with the MTP draft path fully exercised (draft acceptance 0.62), RestartCount=0. No crash.
• --parallel 2: crashes on first decode as described above.

GPU footprint measured after the first decode (once the pp compute buffers are actually reserved), same single 16 GB card:

So b9426 needs essentially the entire 16 GB budget for a single slot — the same envelope in which b9246 fit two. At --parallel 2 the second slot's pp buffer reservation (~983 MiB) has nowhere to go and OOMs. During model load b9426 sits at ~11.9 GB; the jump to ~15.6 GB happens on the first decode when the compute buffers are reserved. This rules out a tight-budget fluke or an allocator bug — drop one slot and b9426 serves cleanly with MTP working. The regression is added per-context compute-buffer footprint in the MTP draft path between b9246 and b9426.

Note on llama-completion/-fit: the crash is in the server's MTP speculative draft path (--spec-type draft-mtp + multiple slots), so it reproduces through llama-server, not llama-completion. The flags above are the minimal repro.

First Bad Commit

Not git-bisected to a single commit. Last known good: b9246 (871b0b7); first confirmed bad: b9426 (1738129).

Prime suspect: #23287 "Move to backend sampling for MTP draft path". The MTP draft init line gained a new field between the two builds:

• b9246: common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00, n_embd=2048
• b9426: common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00, n_embd=2048, backend_sampling=1

Moving draft sampling into the GPU graph would add compute-buffer footprint to the draft context, reserved lazily on first decode — matching both where the crash lands and what the --parallel bisect shows. Other MTP/graph changes in the b9246→b9426 range that could contribute: #23643 (llama: add llm_graph_input_mtp) and #23646 (server: MTP layer kv-cache should respect draft type ctk).

Relevant log output

0.39.130.901 I slot launch_slot_: id  1 | task 0 | processing task, is_child = 0
0.39.850.795 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 982.95 MiB on device 0: cudaMalloc failed: out of memory
0.39.850.801 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 1030694912
0.39.850.802 E graph_reserve: failed to allocate compute buffers
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x23f)
libllama.so.0(llama_decode+0xf)
libllama-common.so.0(common_speculative_impl_draft_mtp::process(llama_batch const&)+0x2df)
libllama-common.so.0(common_speculative_process(...)+0x40)
libllama-server-impl.so(server_context_impl::update_slots()+0x237d)
terminate called after throwing an instance of 'std::runtime_error'
  what():  failed to allocate compute pp buffers

common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00, n_embd=2048, backend_sampling=1
srv  llama_server: server is listening on http://0.0.0.0:8000
srv  update_slots: all slots are idle
slot print_timing: id  0 | task 3 | draft acceptance = 1.00000 (    3 accepted /     3 generated)
slot print_timing: id  0 | task 7 | draft acceptance = 0.61538 (   48 accepted /    78 generated)