spec + vulkan: Performance Drop with MTP n-max > 1 and -np > 1

@0cc4m

Name and Version

from master: 2c6b141

Build / running in docker, arch-base image, host has kernel 7.1.8, RADV/MESA 26.1.7.

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server in docker with compose:
llama-server-vulkan-arch:
    build:
      context: .
      target: server
      dockerfile: .devops/vulkan-arch.Dockerfile
    container_name: llamacpp
    ipc: host
    networks:
      - ai_stack
    ports:
      - "11435:8080"
    environment:
      - GGML_VK_VISIBLE_DEVICES=0,2,1
      - GGML_VK_ALLOW_GRAPHICS_QUEUE=1
    volumes:
      - /mnt/workdrive/llamacpp/models:/models
      - /mnt/workdrive/llamacpp/models_rpc.ini:/app/models.ini
      - /mnt/workdrive/llamacpp/huggingface:/root/.cache/huggingface
    group_add:
      - video
    devices:
      - "/dev/kfd:/dev/kfd"
      - "/dev/dri:/dev/dri"
    entrypoint: ["./llama-server"]
    command: >
      --host 0.0.0.0
      --port 8080
      --models-dir /models
      --models-preset /app/models.ini
      --models-max 2
      --sleep-idle-seconds 6000
      --metrics
      --load-mode none
      --offline
      --log-verbosity 3
      --sse-ping-interval 15
    restart: unless-stopped


Model ini:
qwen3-8-27b]
cache-ram              = 14336
ctx-checkpoints        = 4
checkpoint-min-step    = 8192
main-gpu               = 0
parallel               = 3
batch-size             = 4096
ubatch-size            = 512
kv-unified             = false
spec-type              = draft-mtp
spec-draft-n-max       = 1
spec-draft-n-min       = 0
hf                     = unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
ctx-size               = 491520
temp                   = 1.0
top-p                  = 0.95
top-k                  = 20
min-p                  = 0.0
presence-penalty       = 0.0
repeat-penalty         = 1.0
cache-type-k           = q8_0
cache-type-v           = q8_0
flash-attn             = true
split-mode             = layer
tensor-split           = 50,50,0
jinja                  = true
reasoning-preserve     = true
reasoning-effort       = medium
reasoning-budget       = 20000
image-min-tokens       = 1024
no-mmproj-offload      = true

Problem description & steps to reproduce

On vulkan with np>1 and actually dispatching parallel sessions, along with model-native MTP (tested with Q3.6-27B and Q3.8-27B, different quants, from UD-Q4-XL to Q6_K) the performance collapses when n-max for the draft is set to > 1. My setup has 2xR9700, and ROCM seems to scale better, especially for np > 2, but I only tested until np = 4. See below. Single session works fine as expected, and also scales well with n-max, depending on the workload of course. With very large n-max (5-6) i sometime even see a spike to 60TPS+ when the predictions hit the spot. But mostly its in the 42-46TPS range single session. Setup / Contig notes included, mind that I rund with large context, 3x160k(*1024).

This has also been reported in other issues, but I want to collect this here, so we can maybe focus efforts, debugging, etc. to arrive at a root cause. @0cc4m this follows up my statements in the dflash2 PR; hope this is a better home and naming you here since you mentioned that you are not aware of an issue with the scaling currently. Nvidia does not seem to be affected, but AMD seems to be, and one report in the dflash2 PR #27342 from one intel user did show the same behavior. I noticed that MTP/Dflash2 had the same issue, so my speculation is somewhere else, either in the vulkan backend or in some interworking between the sessions, spec and the backend.

I tried to seach around, and here are some issues which might be related and/or have content / benchmarks which show the issue.
#26360 (Feature Request Vulkan)
#23774 (MTP Perf Deg Vulkan)
#24066 (General Vulkan Perf drop)
#26663 (Vulkan Ops Profiles RDNA4)
#27181 (Helpful links to other Projects, also RDNA4 specific)
#21043 (Lots of performance measurements, also some with the symptom, but also without, RDNA4)

I'm happy to assist with any kind of debug output needed, and test patches. Would be super cool to have scalability with vulkan as well. Since I saw the same issue when testing the dFlash2 draft early on, this might be benefical there as well. If any further info is needed, dont hesitate.

Handwritten, Only Benchmark summary from AI summarized benchmark doc.

First Bad Commit

Unkown currently

Relevant log output

Logs

Derived batch widths (N · L, L = 1 + n_max):

row	np=1	np=2	np=4
no spec (L=1)	1	2	4
MTP n-max 2 (L=3)	3	6	12

Sweep summary (raw)

run	`n_max`	L	N	N·L	single TG	parallel TG (sum)	parallel/single	single acc	parallel acc (range)
6a	1	2	2	4	33.72	42.96	1.27×	0.644	0.661 – 0.686
1	1	2	3	6	35.11	52.65	1.50×	0.699	0.656 – 0.780
4	1	2	3	6	34.07	55.78	1.64×	0.693	0.643 – 0.675
6b	3	4	2	8	35.42	41.12	1.16×	0.416	0.415 – 0.425
5	1	2	4	8	33.87	55.58	1.64×	0.632	0.632 – 0.683
3	2	3	3	9	35.95	38.52	1.07×	0.529	0.530 – 0.560
2	3	4	3	12	37.52	40.78	1.09×	0.458	0.393 – 0.500

Runs 6b and 5 are the same batch width (8 tokens) with different L — 4 vs 2 — and differ by
1.16× vs 1.64×. Recorded here without interpretation.

Per-position acceptance, all verbosity-4 runs:

run	L	single	parallel
4	2	(0.693)	(0.643), (0.675), (0.645)
5	2	(0.632)	(0.672), (0.683), (0.656), (0.657)
6a	2	(0.644)	(0.686), (0.661)
3	3	(0.674, 0.383)	(0.662, 0.398), (0.675, 0.437), (0.727, 0.393)
6b	4	(0.672, 0.362, 0.213)	(0.724, 0.342, 0.179), (0.676, 0.395, 0.205)

Mean accepted length: run 1 1.70/1.66–1.78 · run 4 1.69/1.64–1.67 · run 5 1.63/1.66–1.68 ·
run 3 2.06/2.06–2.12 · run 2 2.37/2.18–2.50.

Runs 1 and 4 are the same configuration measured twice (4 with verbosity 4). Run-to-run spread on
the headline number: parallel TG 52.65 vs 55.78 (±6%), single TG 35.11 vs 34.07 (±3%).

The verify batch is N · L tokens wide. The threshold is 8.

configuration	N·L	path	observed
no spec, `-np 3`	3	vec	linear scaling ✓
spec L=2, `-np 1`	2	vec	large win ✓
spec L=4, `-np 1`	4	vec	large win ✓
spec L=2, `-np 3`	6	vec	still scales, 1.64× ✓
spec L=3, `-np 3`	9	GEMM	collapse, 1.07× ✓
spec L=4, `-np 3`	12	GEMM	collapse, 1.09× ✓

ROCM

Run R1 — `spec-draft-n-max = 1` (L = 2), `parallel = 3`

R1a — single (slot 2, task 0)

metric	value
prompt eval	357.63 ms / 14 tok — 25.55 ms/tok — 39.15 t/s
eval	17343.49 ms / 563 tok — 30.86 ms/tok — 32.40 t/s
graphs reused	327
draft acceptance	0.70821 (233/329), mean len 1.71
acc per pos	(0.708)

R1b — 2 parallel (tasks 332, 333)

slot	task	eval	ms/tok	t/s	graphs reused	acceptance	mean len
2	333	21937.01 ms / 498 tok	44.14	22.66	606	0.762 (215/282)	1.76
1	332	22918.86 ms / 503 tok	45.66	21.90	626	0.657 (199/303)	1.66

R1c — 3 parallel (tasks 639, 640, 641)

slot	task	eval	ms/tok	t/s	graphs reused	acceptance	mean len
0	641	23564.45 ms / 400 tok	59.06	16.93	860	0.691 (163/236)	1.69
2	639	23807.62 ms / 393 tok	60.73	16.47	860	0.658 (156/237)	1.66
1	640	24874.73 ms / 426 tok	58.53	17.09	879	0.658 (169/257)	1.66

Aggregate TG: single 32.40 → dual 44.56 (1.38×) → triple 50.49 (1.56×).
Tick time (ms/tok × mean_len): width 2 = 52.8 ms · width 4 = 76.8 ms · width 6 = 99.3 ms.

Vulkan comparison at the same config (Runs 1/4): single 34–35 t/s, triple 52.7–55.8 t/s (1.50–1.64×),
tick time 49.7 ms (width 2) → 85.5–89.3 ms (width 6). ROCm tracks the Vulkan collapse pattern
closely here — same order of scaling ratio, ~10–15% higher absolute tick cost at width 6.

Run R2 — `spec-draft-n-max = 3` (L = 4), `parallel = 3`

R2a — single (slot 2, task 0)

metric	value
prompt eval	300.74 ms / 14 tok — 21.48 ms/tok — 46.55 t/s
eval	21648.87 ms / 712 tok — 30.45 ms/tok — 32.84 t/s
graphs reused	276
draft acceptance	0.52278 (436/834), mean len 2.57
acc per pos	(0.719, 0.496, 0.353)

R2b — 2 parallel (tasks 281, 282)

slot	task	eval	ms/tok	t/s	graphs reused	acceptance	mean len
2	281	22204.66 ms / 374 tok	59.53	16.80	445	0.393 (203/516)	2.18
1	282	27441.32 ms / 505 tok	54.45	18.37	514	0.365 (265/726)	2.10

R2c — 3 parallel (tasks 527, 528, 529)

slot	task	eval	ms/tok	t/s	graphs reused	acceptance	mean len
0	529	17070.45 ms / 455 tok	37.60	26.60	703	0.462 (265/573)	2.39
2	528	20411.33 ms / 461 tok	44.37	22.54	726	0.381 (247/648)	2.14
1	527	20822.56 ms / 465 tok	44.88	22.28	730	0.367 (243/663)	2.10

Aggregate TG: single 32.84 → dual 35.17 (1.07×) → triple 71.42 (2.18×).
Tick time: width 4 = 78.3 ms · width 8 = 122.0 ms · width 12 = 93.4 ms.

Sidenote: I tried to start with GGML_VK_PERF_LOGGER=1 and GGML_VK_PERF_LOGGER_FREQUENCY=99999, which resulted in model load failure (latest commit used, b21e4de) with:

0.14.693.300 I srv  ensure_model: waiting until model name=qwen3-8-27b is fully loaded...
[41075] WARNING: radv is not a conformant Vulkan implementation, testing use only.
[41075] WARNING: radv is not a conformant Vulkan implementation, testing use only.
[41075] warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
[41075] 0.00.045.563 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
[41075] 0.00.046.254 W srv  llama_server: -----------------
[41075] 0.00.046.265 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
[41075] 0.00.046.265 W srv  llama_server: this can be a security risk (cross-origin attacks)
[41075] 0.00.046.265 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
[41075] 0.00.046.265 W srv  llama_server: -----------------
[41075] 0.00.047.627 I srv    load_model: loading model 'unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL'
[41075] 0.04.141.729 W cmn  set_process_: failed to set process priority 2 : Permission denied (13)
[41075] 0.04.141.742 I cmn          init: llama threadpool init, n_threads = 24
[41075] /app/ggml/src/ggml-vulkan/ggml-vulkan.cpp:17100: GGML_ASSERT(ctx->compute_ctx.expired()) failed
[41075] libggml-base.so.0(+0x19305) [0x7f64f5708305]
[41075] libggml-base.so.0(ggml_print_backtrace+0x200) [0x7f64f5708800]
[41075] libggml-base.so.0(ggml_abort+0x12e) [0x7f64f57089ce]
[41075] /app/libggml-vulkan.so(+0x315b67) [0x7f64f05b1b67]
[41075] libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x847) [0x7f64f5726f37]
[41075] libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa0) [0x7f64f58c8b60]
[41075] libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x114) [0x7f64f58cd154]
[41075] libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x4bd) [0x7f64f58d1c0d]
[41075] libllama.so.0(llama_decode+0xe) [0x7f64f58d372e]
[41075] libllama-common.so.0(_Z23common_init_from_paramsR13common_paramsb+0x3a1) [0x7f64f5f8dc01]
[41075] libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0x5d1) [0x7f64f63a56e1]
[41075] libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x3f6e) [0x7f64f62f476e]
[41075] libllama-server-impl.so(_Z12llama_serveriPPc+0x1151) [0x7f64f62f69d1]
[41075] /usr/lib/libc.so.6(+0x27781) [0x7f64f4a1a781]
[41075] /usr/lib/libc.so.6(__libc_start_main+0x89) [0x7f64f4a1a8b9]
[41075] /app/llama-server(+0x1075) [0x55fd34ce4075]

Another ref with nearly the same issue, but with rocm: #27117
Although it is a different backend, its strange that it shows almost the same behave, while CUDA is seemingly unnafected

Same pattern here on a different AMD setup. In addition to the sweep in the issue: the n-max 1 (width 8) vs n-max 2/3 (width 12/16) split at 4 slots on five dense 27B GGUFs and one MoE, and the batch shape without speculation. One box, one backend (Vulkan/RADV), so nothing here says anything about ROCm or CUDA.

Setup: Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151), 128 GiB unified memory, kernel 7.0.0-38-generic, Mesa 26.0.8 RADV, llama.cpp master c06f84160 (build 11427). llama-server with --parallel 4 --kv-unified -fa on -ub 512 -c 262144 --spec-type draft-mtp. Client: concurrent chat requests, 512 generated tokens each, ignore_eos, enable_thinking: true, no sampler keys in the request. Server sampling on the Qwen GGUFs: temp 1.0, top-p 0.95, top-k 20, min-p 0.0, presence penalty 0.0, --reasoning on. Numbers are aggregate t/s, median of the rounds.

Dense 27B, 4 streams vs the same model at 4 streams without a drafter (6 rounds per arm over two runs about 19 h apart, same build and args; width = 4 × (1 + n-max)):

GGUF	no drafter	n-max 1 (width 8)	served n-max (width)
`Qwen3.8-27B-UD-IQ4_XS.gguf` + `mtp-Qwen3.8-27B-Q4_0.gguf`	42.02	38.62 ×0.92	3 (16): 30.38 ×0.72
`Swift-Qwen3.8-27B-Q4_K_M.gguf`	36.35	36.36 ×1.00	3 (16): 27.00 ×0.74
`Qwen3.8-27B-TurboFCFusion-735-882-Here-Uncen-NEO-CODER-MAX-MTP-Q6_K.gguf`	26.70	30.05 ×1.13	2 (12): 24.71 ×0.93
`Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf`	46.15	36.78 ×0.80	2 (12): 29.83 ×0.65
`Qwen3.6-27B-IQ4_XS.gguf`	41.08	42.58 ×1.04	3 (16): 35.65 ×0.87

The MTP head is inside the GGUF except in row 1. KV is q4_0 except the Q6_K row (f16). On all five, each of the 6 n-max 1 rounds was faster than every one of the 6 rounds at width 12/16.

MoE: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf (512 experts, 10 used) + mtp-Qwen3.8-Flash-Next-Q8_0.gguf (6 runs / 6 rounds each):

	no drafter	n-max 1	n-max 3
1 stream (width 1 / 2 / 4)	27.91	33.27 ×1.19	30.38 ×1.09
4 streams (width 4 / 8 / 16)	54.69	48.61 ×0.89	22.01 ×0.40

Same MoE, no speculation: llama-batched-bench -c 16384 -npp 512 -ntg 128 -npl 1,4,8,9,12,16 -kvu -fa on -ngl 999, median S_TG of 3 runs: B1 28.18, B4 64.75, B8 78.23, B9 28.37, B12 36.65, B16 46.65. I have not run batched-bench on the dense models.

#27332 (tested as merge 373ada9e7; the newer head only adds an NVIDIA cap), same MoE: batched-bench B9 / B12 / B16 +8.4 / +10.0 / +8.6 %, B1 / B4 / B8 +0.1 / −0.2 / −0.2 %. MTP at 4 streams: n-max 1 48.61 → 49.47, n-max 3 22.01 → 23.31, no drafter 54.69 → 54.66 (separate runs about 9 h apart; the min-max ranges of the MTP arms overlap). On the PR build every n-max 1 and n-max 3 round at 4 streams is still below every no-drafter round, so on this box it does not change the 4-slot picture. The PR only changes the MUL_MAT_ID gate, so the dense rows are outside it (not re-run).

This is consistent with the 8-column mat-vec limit at c06f84160: mul_mat_vec_max_cols = 8 is used for MUL_MAT at ggml-vulkan.cpp:7296, and MUL_MAT_ID checks src2->ne[1] <= 8 at ggml-vulkan.cpp:7918. I have not profiled it, so this is a fit, not a proof.

One exception: gemma-4-26B-A4B-it-Q8_0.gguf (MoE, 128 experts, 8 used) with its separate assistant drafter gemma-4-26B-A4B-it-assistant.Q8_0.gguf (server args: top-k 64, --reasoning off) gained at 4 streams. No drafter: 84.37; n-max 1: 92.15 ×1.09; n-max 3 (width 16): 115.52 ×1.37.

Follow-up: single-stream numbers for the same five dense GGUFs, so the trade is visible. Same box and build (c06f84160), same server args, 512 generated tokens. One stream: 3 runs per drafter arm, 6 for no drafter (bookended: a no-drafter start before and after the drafter arms). The 4-stream ratios are from the same run, and their direction matches the table above.

GGUF	1 stream: no drafter	n-max 1	served n-max	4 streams vs no drafter: n-max 1 / served
`Qwen3.8-27B-UD-IQ4_XS.gguf` + `mtp-Qwen3.8-27B-Q4_0.gguf`	14.91	20.41 ×1.37	3: 25.27 ×1.69	×0.92 / ×0.72
`Swift-Qwen3.8-27B-Q4_K_M.gguf`	12.33	17.09 ×1.39	3: 21.12 ×1.71	×1.00 / ×0.71
`Qwen3.8-27B-TurboFCFusion-735-882-Here-Uncen-NEO-CODER-MAX-MTP-Q6_K.gguf`	9.29	13.62 ×1.47	2: 15.51 ×1.67	×1.14 / ×0.98
`Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf`	18.82	26.31 ×1.40	2: 28.95 ×1.54	×0.78 / ×0.67
`Qwen3.6-27B-IQ4_XS.gguf`	13.82	21.16 ×1.53	3: 26.19 ×1.89	×1.03 / ×0.86

So on this box the longer draft wins alone (×1.54-1.89 vs ×1.37-1.53 by median) and loses with 4 slots busy; n-max 1 (verify width 8) is at or above no-drafter at 4 streams on three of the five.

Correction to the gemma line in my comment above: the client sent enable_thinking: true, which overrides the server's --reasoning off, so that row was measured with thinking on.

One more datapoint, since I said above I had not run batched-bench on a dense model. Same box and build (c06f84160), no speculation:

llama-batched-bench -m Qwen3.8-27B-UD-IQ4_XS.gguf -c 16384 -npp 512 -ntg 128 -npl 1,4,8,9,12,16 -kvu -fa on -ngl 999 -ctk q4_0 -ctv q4_0

Median S_TG of 3 runs (run-to-run range ≤ 0.3 %): B1 14.96, B4 45.89, B8 61.72, B9 40.83, B12 50.08, B16 60.27.

So the dense model also steps down past 8 sequences: B9/B8 ≈ ×0.66 (the MoE above: ×0.36), and B16 is still below B8.

Navigation Menu

Repository navigation

spec + vulkan: Performance Drop with MTP n-max > 1 and -np > 1 #27544

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Sweep summary (raw)

Run R1 — `spec-draft-n-max = 1` (L = 2), `parallel = 3`

Run R2 — `spec-draft-n-max = 3` (L = 4), `parallel = 3`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

spec + vulkan: Performance Drop with MTP n-max > 1 and -np > 1 #27544

Description

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Sweep summary (raw)

Run R1 — spec-draft-n-max = 1 (L = 2), parallel = 3

Run R2 — spec-draft-n-max = 3 (L = 4), parallel = 3

Activity

Stoney49th commented on Aug 22, 2026

Stoney49th commented on Aug 23, 2026

AIalliAI commented on Oct 6, 2026

AIalliAI commented on Oct 7, 2026

AIalliAI commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Run R1 — `spec-draft-n-max = 1` (L = 2), `parallel = 3`

Run R2 — `spec-draft-n-max = 3` (L = 4), `parallel = 3`