Skip to content

Eval bug: Severe speed drop from 50 t/s to 30 t/s in recent updates on RTX 5060 Ti #139

Closed
Closed
Eval bug: Severe speed drop from 50 t/s to 30 t/s in recent updates on RTX 5060 Ti#139

Description

@ethernidee
ethernidee
opened on Aug 28, 2026

Name and Version

beellama v0.4.4
commit 287da7c (HEAD -> v0.4.4, origin/v0.4.4)
Author: Anbeeld anbeeld@gmail.com
Date: Fri Aug 28 01:25:13 2026 +0200
Update CHANGELOG.md

Operating systems

Windows

GGML backends

CUDA

Hardware

Ryzen 7 5700X + RTX 5060 Ti 16GB + 128 GB RAM

Models

Qwen3.8-27B-UD-Q3_K_XL.gguf by unsloth, tried IQ4 and mixed quants too.

Problem description & steps to reproduce

I had 50-55 t/s MTP speed for Qwen 3.8 27B. At least on earlier 0.4.3 and 0.4.4 version. After recent update I got 30-33 t/s both with MTP and dflash2, with kvarn6, kvarn4, kvarn8, q6_0 KV cache.

e:\beellama\build\bin\Release\llama-server.exe ^
  -m "e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf" ^
  --alias "Qwen3.8 27B MTP" ^
  --host 127.0.0.1 --port 8001 ^
  --n-gpu-layers 999 ^
  --gpu-layers-draft 999 ^
  --ctx-size 60000 ^
  --fit off ^
  --min-p 0.0 ^
  --temp 0.6 --top-p 0.95 --top-k 20 ^
  --repeat-penalty 1.03 ^
  -ctk q6_0 -ctv q6_0 --kv-tail-tokens 1024 ^
  --flash-attn on ^
  --batch-size 128 --ubatch-size 128 ^
  --threads 8 --threads-batch 8 ^
  --no-mmap --mlock ^
  --spec-type draft-mtp --spec-draft-n-max 3 ^
  --parallel 1 ^
  --log-colors off ^
  --kv-unified ^
  --metrics ^
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
  --jinja

Built via:

cmake .. ^
  -DCMAKE_BUILD_TYPE=Release ^
  -DGGML_CUDA=ON ^
  -DGGML_NATIVE=ON ^
  -DGGML_CUDA_GRAPHS=ON ^
  -DGGML_CUDA_F16=ON ^
  -DGGML_CUDA_NO_PEER_COPY=ON ^
  -DGGML_CUDA_FP4=ON ^
  -DGGML_CUDA_FA_ALL_QUANTS=ON ^
  -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build . --config Release --target llama-server llama-quantize llama-bench llama-perplexity --parallel

First Bad Commit

No response

Relevant log output

Logs 
e:\llama.cpp\build\bin\Release>e:\beellama\build\bin\Release\llama-server.exe   -m "e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf"   --alias "Qwen3.8 27B MTP"   --host 127.0.0.1 --port 8001   --n-gpu-layers 999   --gpu-layers-draft 999   --ctx-size 60000   --fit off   --min-p 0.0   --temp 0.6 --top-p 0.95 --top-k 20   --repeat-penalty 1.03   -ctk q6_0 -ctv q6_0 --kv-tail-tokens 1024   --flash-attn on   --batch-size 128 --ubatch-size 128   --threads 8 --threads-batch 8   --no-mmap --mlock   --spec-type draft-mtp --spec-draft-n-max 3   --parallel 1   --log-colors off   --kv-unified   --metrics   --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"   --jinja
�[34m0.00.034.806�[0m �[35mW DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead
�[0m�[34m0.00.034.812�[0m �[35mW DEPRECATED: --mlock is deprecated. use --load-mode mlock instead
�[0m0.00.035.438 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.035.672 W srv  llama_server: -----------------
0.00.035.676 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.035.681 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.035.683 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.035.684 W srv  llama_server: -----------------
0.00.044.843 I srv    load_model: loading model 'e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf'
0.05.230.187 I cmn          init: llama threadpool init, n_threads = 8
0.05.334.566 I common_speculative_init_result: creating MTP draft context against the target model 'e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf'
0.05.345.758 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 60160, kv_unified = 'true'
0.05.352.932 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.05.352.946 I srv  llama_server: model loaded
0.05.352.951 I srv  llama_server: listening on http://127.0.0.1:8001
0.16.253.613 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.16.253.729 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.20.303.828 I slot print_timing: id  0 | task 0 | n_gen =    103, tg =  33.98 t/s, tg_3s =  34.31 t/s
0.23.372.525 I slot print_timing: id  0 | task 0 | n_gen =    202, tg =  33.11 t/s, tg_3s =  32.26 t/s
0.26.438.205 I slot print_timing: id  0 | task 0 | n_gen =    305, tg =  33.27 t/s, tg_3s =  33.60 t/s
0.29.485.206 I slot print_timing: id  0 | task 0 | n_gen =    405, tg =  33.16 t/s, tg_3s =  32.82 t/s
0.32.531.383 I slot print_timing: id  0 | task 0 | n_gen =    506, tg =  33.16 t/s, tg_3s =  33.16 t/s
0.35.558.491 I slot print_timing: id  0 | task 0 | n_gen =    598, tg =  32.70 t/s, tg_3s =  30.39 t/s
0.38.616.017 I slot print_timing: id  0 | task 0 | n_gen =    691, tg =  32.37 t/s, tg_3s =  30.42 t/s
0.41.670.634 I slot print_timing: id  0 | task 0 | n_gen =    800, tg =  32.79 t/s, tg_3s =  35.68 t/s
0.44.699.564 I slot print_timing: id  0 | task 0 | n_gen =    901, tg =  32.85 t/s, tg_3s =  33.35 t/s
0.47.768.127 I slot print_timing: id  0 | task 0 | n_gen =    989, tg =  32.43 t/s, tg_3s =  28.68 t/s
0.50.821.897 I slot print_timing: id  0 | task 0 | n_gen =   1082, tg =  32.25 t/s, tg_3s =  30.45 t/s
0.53.831.208 I slot print_timing: id  0 | task 0 | n_gen =   1165, tg =  31.87 t/s, tg_3s =  27.58 t/s
0.56.865.386 I slot print_timing: id  0 | task 0 | n_gen =   1247, tg =  31.49 t/s, tg_3s =  27.03 t/s
0.59.913.022 I slot print_timing: id  0 | task 0 | n_gen =   1362, tg =  31.94 t/s, tg_3s =  37.73 t/s
1.02.936.742 I slot print_timing: id  0 | task 0 | n_gen =   1455, tg =  31.86 t/s, tg_3s =  30.76 t/s
1.05.999.930 I slot print_timing: id  0 | task 0 | n_gen =   1575, tg =  32.32 t/s, tg_3s =  39.17 t/s
1.09.055.718 I slot print_timing: id  0 | task 0 | n_gen =   1697, tg =  32.77 t/s, tg_3s =  39.92 t/s
1.12.060.849 I slot print_timing: id  0 | task 0 | n_gen =   1833, tg =  33.46 t/s, tg_3s =  45.26 t/s
1.12.850.473 W srv          stop: cancel task, id_task = 0
1.12.914.335 I slot      release: id  0 | task 0 | stop processing: n_tokens = 2263, truncated = 0

Activity

  1. Anbeeld commented on Aug 28, 2026

    @Anbeeld
    Anbeeld
    on Aug 28, 2026
    Owner

    Thanks, this looks like a real performance regression.

    I found a potentially related upstream report under the DFlash2 PR: on an RTX 5080, MTP decode dropped from 183.7 to ~160 t/s with essentially unchanged acceptance. They traced it to changes on the DFlash2 branch itself, not an upstream rebase, and already ruled out the new top-k kernel, draft sampling, "p_min" behavior, and CUDA graphs.

    Could you A/B these official upstream Windows CUDA 13.3 builds with the same target model, prompt/workload and MTP settings? Use standard upstream KV since KVarN/tail flags don't exist there.

    Please post generation t/s + acceptance for each.

    This should narrow it down pretty cleanly:

    • b10644 fast, b10657 slow → regression is in upstream changes before the DFlash2 merge.
    • b10657 fast, b10658 slow → very likely the same DFlash2-branch/shared speculative regression.
    • all three upstream builds fast → likely BeeLlama-specific integration or KV precision-tail path.

    Current hypothesis is a shared graph/backend execution regression. The upstream Blackwell report is especially suspicious because it affects MTP with unchanged acceptance, although their measured regression was smaller and NVFP4-specific, so it may only be part of what we're seeing here.

    Upstream investigation: ggml-org#27342 (comment)

  2. Anbeeld commented on Aug 28, 2026

    @Anbeeld
    Anbeeld
    on Aug 28, 2026
    Owner

    One more useful test: please compare the release build against a local MSVC + CUDA build from the exact same BeeLlama commit.

    There is a related upstream Blackwell issue where the official Windows CUDA binaries were much slower, but rebuilding the same/newer source with MSVC + CUDA completely removed the regression: ggml-org#26918 (comment)

    BeeLlama's current Windows release pipeline follows a similar split to upstream: the Windows executable/CPU side is built with the LLVM toolchain, while "ggml-cuda.dll" is built separately with CUDA.

    So if possible, please test:

    • current "v0.4.4" release binary
    • the exact same "v0.4.4" commit built locally with normal MSVC + CUDA

    Same model, prompt and flags.

    If the local MSVC build restores the old ~50 t/s performance, that would point at the Windows release toolchain rather than the DFlash2/KV changes themselves.

  3. ethernidee commented on Aug 28, 2026

    @ethernidee
    ethernidee
    on Aug 28, 2026
    Author

    Ok, thanks, I'm trying different configurations currently.

  4. Anbeeld commented on Aug 28, 2026

    @Anbeeld
    Anbeeld
    on Aug 28, 2026
    Owner

    Also found a much closer upstream match to your report:

    ggml-org#25489 (comment)

    That user is also on an RTX 5060 Ti 16 GB and reports that current llama.cpp MTP performance drops hard once context reaches ~60k+, averaging around 30 t/s. Rebuilding an older upstream commit (8a091c47abe67e0a03b85bc7c9eee8bdb9b14b05) restores ~41 t/s average and much more stable performance.

    Importantly, these are locally built binaries, so this regression exists independently of the Windows release build/toolchain issue mentioned above.

    Could you also try building upstream commit: 8a091c47abe67e0a03b85bc7c9eee8bdb9b14b05 with MSVC + CUDA and run the same ~60k-context MTP workload?

    If that restores the old performance on your 5060 Ti too, this is very likely the same upstream regression and gives us a much better range to bisect.

  5. ethernidee commented on Aug 28, 2026

    @ethernidee
    ethernidee
    on Aug 28, 2026
    Author

    By the way, it seemed to be a bug in upstream DFlash2 implementation, causing hard speed drop when using tool calls.
    There is a fork with bugfix: https://github.com/HermiG/llama.cpp/tree/fix/dflash2-tool-tg-collapse

    fix: avoid DFlash2 grammar scan collapse with tool calls

    common : add --spec-draft-ubatch-size argument to control block draft…

  6. Anbeeld commented on Aug 28, 2026

    @Anbeeld
    Anbeeld
    on Aug 28, 2026
    Owner

    By the way, it seemed to be a bug in upstream DFlash2 implementation, causing hard speed drop when using tool calls. There is a fork with bugfix: https://github.com/HermiG/llama.cpp/tree/fix/dflash2-tool-tg-collapse

    fix: avoid DFlash2 grammar scan collapse with tool calls

    common : add --spec-draft-ubatch-size argument to control block draft…

    Have you tried it, does it resolve the issue for you?

  7. Anbeeld commented on Aug 28, 2026

    @Anbeeld
    Anbeeld
    on Aug 28, 2026
    Owner

    One clarification after looking into those commits more closely: that HermiG branch is based on an earlier DFlash2 development path, not the final DFlash2 implementation that was merged upstream and then into BeeLlama.

    So I would not cherry-pick those commits directly yet.

    What may still be very relevant is the trigger they identified: tool calls / active grammar causing a large TG collapse.

  8. ethernidee commented on Aug 28, 2026

    @ethernidee
    ethernidee
    on Aug 28, 2026
    Author

    Sorry, seems I was wrong making short tests in a hurry. The speed starts at 30-35 t/s, but later growth on non-thinking and approaches 47 t/s.
    I made multiple tests with upstream llama and latest beellama, everything rebuilt from scratch with CUDA_ARCHITECTURE=120. So the issue may be closed, I suppose.
    Here are the results:

    SYSTEM PROMPT

Write a single-file HTML/JS/CSS Windows-style desktop application with six apps, a start menu, a task manager, and a console.
No external dependencies. The whole desktop and each application must be ui friendly ,beautiful, fully functional,
meaningful and useful, including all tabls and settings.

------------------------------

llama-server-upstream.exe ^
  -m "e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf" ^
  --alias "Qwen3.8 27B MTP" ^
  --host 127.0.0.1 --port 8001 ^
  --n-gpu-layers 999 ^
  --ctx-size 60000 ^
  --fit off ^
  --min-p 0.0 ^
  --temp 0.6 --top-p 0.95 --top-k 20 ^
  --repeat-penalty 1.03 ^
  -ctk q4_1 -ctv q4_1 ^
  --flash-attn on ^
  --batch-size 128 --ubatch-size 128 ^
  --threads 8 --threads-batch 8 ^
  --no-mmap --mlock ^
  --spec-type draft-mtp --spec-draft-n-max 3 ^
  --parallel 1 ^
  --log-colors off ^
  --kv-unified ^
  --metrics ^
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
  --jinja

17571 tokens ; 6min 10s ; 47.45 t/s ; draft acceptance = 0.70436 ( 9175 accepted / 13026 generated), mean len =  3.11

------------------------------

[NON-MTP] beellama-server.exe ^
  -ctk q4_1 -ctv q4_1
  ...

17002 tokens; 11min 15s ; 25.17 t/s (note, this is NON-MTP run)

------------------------------

beellama-server.exe ^
  -ctk q4_1 -ctv q4_1
  ...

17452 tokens; 5.92 min ; 49.13 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn8 -ctv kvarn6 --kv-tail-tokens 1024 ^
  ...

19453 tokens ; 7min 24s ; 43.80 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn6 -ctv kvarn4 --kv-tail-tokens 1024 
  ...

19302 tokens ; 8min 25s ; 38.21 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn6 -ctv kvarn4 --kv-tail-tokens 1024 
  ...

16410 tokens; 6min 10s ; 44.34 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn4 -ctv kvarn4 --kv-tail-tokens 1024 ^
  ...

15963 tokens ; 5min 59s ; 44.37 t/s

------------------------------

beellama-server.exe ^
  -md "Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf" ^
  --spec-type draft-dflash --spec-draft-n-max 5 ^
  -ctk q4_1 -ctv q4_1 --kv-tail-tokens 1024 ^

15789 tokens; 5min 30s; 47.77 t/s ; draft acceptance = 0.48516 (11178 accepted / 23040 generated), mean len =  3.43

------------------------------

beellama-server.exe ^
  -md "Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf" ^
  --spec-type draft-dflash --spec-draft-n-max 4 ^
  -ctk q4_1 -ctv q4_1 --kv-tail-tokens 1024 ^

14121 tokens; 5min 4s; 46.30 t/s ; draft acceptance = 0.57774 ( 9854 accepted / 17056 generated), mean len =  3.31

------------------------------

beellama-server.exe ^
  -md "Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf" ^
  --spec-type draft-dflash --spec-draft-n-max 6 ^
  -ctk q4_1 -ctv q4_1 --kv-tail-tokens 1024 ^

15315 tokens; 5min 32s; 46.06 t/s ; draft acceptance = 0.40785 (10870 accepted / 26652 generated), mean len =  3.45

  9. Anbeeld commented on Aug 29, 2026

    @Anbeeld
    Anbeeld
    on Aug 29, 2026
    Owner

    No problem, thank you for keeping an eye on Bee!

  10. Anbeeld
    closed this as completedon Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions