Skip to content

Eval bug: Gemma4 MTP #24350

Closed as not planned
#24590
Closed as not planned
Eval bug: Gemma4 MTP#24350
#24590
Labels
bug-unconfirmedstale

Description

@b1231227
b1231227
opened on Jun 9, 2026

Name and Version

version: 9573 (f0152ef)

Operating systems

Windows

GGML backends

CUDA

Hardware

2*3060 12G

Models

gemma-4-12B-it-heretic-QAT-UD-Q4_K_XL.gguf
https://huggingface.co/SC117/gemma-4-12B-it-heretic-QAT-GGUF

Problem description & steps to reproduce

@echo off
set CUDA_DEVICE_ORDER=PCI_BUS_ID

start /affinity FFF0 /b /wait F:\ai_system\llama-cpp-cuda\llama-server.exe ^
-m F:\ai_models\gemma4\12B\gemma-4-12B-it-heretic-QAT-UD-Q4_K_XL.gguf ^
--spec-draft-model F:\ai_models\gemma4\12B\MTP\gemma-4-12B-it-qat-assistant-MTP-ONLY_Q4_0.gguf ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
-ngl 99 ^
-ts 50,50 ^
--threads 12 ^
-c 204800 ^
-b 1024 ^
-ub 128 ^
-np 1 ^
-ctk q8_0 ^
-ctv q8_0 ^
-fa on ^
--temp 1 ^
--top-k 64 ^
--top-p 0.95 ^
--min-p 0.00 ^
--presence_penalty 1.0 ^
--repeat-penalty 1.0^
--frequency-penalty 0.05 ^
--xtc-probability 0 ^
--mirostat 0 ^
--samplers "top_k;top_p;min_p;temperature" ^
--reasoning on ^
--seed -1 ^
--host 0.0.0.0 ^
--port 8000 ^
--jinja ^
--chat-template-file F:\ai_models\gemma4\12B\custom_pub_chat_template_gemma4.jinja ^
--ctx-checkpoints 2 ^
--warmup ^
--mlock ^
--no-mmap ^
---fit on ^
-v

But there is no problem when fit off.

First Bad Commit

No response

Relevant log output

0.01.198.364 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
0.01.264.271 W srv load_model: [spec] failed to measure draft model memory: failed to create llama_context from model

Activity

  1. b1231227
    added
    bug-unconfirmed
    on Jun 9, 2026
  2. leotm
    added a commit that references this issue on Jun 13, 2026

    llama : suppress misleading Gemma4Assistant error during memory fitting

    49138c8

  3. alexhalf commented on Jun 14, 2026

    @alexhalf
    alexhalf
    on Jun 14, 2026

    rxt3060 12gb gemma4-12b-it-qat without mtp give me 31-36 t/s
    with mtp start of 52 to 16 t/s with context from 0 to 1365
    and I have decrease context from 262144 to 190000 and set --no-kv-offload to run server else exit
    may be i need more vram :(?

    "gemma-4-12B":
    cmd: |
      /models/llm/llama-b9631/bin/helper.sh --port 8400 --model /models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf    
      --spec-type draft-mtp --spec-draft-n-max 4 --model-draft /models/mtp-gemma-4-12B-it.gguf --spec-draft-cpu-moe --no-kv-offload
      --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64
      --reasoning off
      -ngl 99
      --ctx-size 190000
      --batch-size 2048
      --ubatch-size 1024
      --flash-attn on
      --jinja
      --metrics
      --mlock
      --no-mmap
      --cache-type-k q8_0
      --cache-type-v q4_0
      --parallel 1
      --cont-batching
      --temp 0.7
      --top-p 0.95
      --top-k 40
      --kv-unified
      --cache-idle-slots
      --slot-save-path /models/cache/gemma4-12b
      --ctx-checkpoints 1024
    proxy: http://127.0.0.1:8400
    0.00.033.593 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.033.609 I device_info:
0.00.194.896 I   - CUDA0   : NVIDIA GeForce RTX 3060 (12287 MiB, 11242 MiB free)
0.00.194.926 I   - CPU     : AMD Ryzen 9 7900X 12-Core Processor (63490 MiB, 63490 MiB free)
0.00.195.060 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.195.248 I srv          init: using 23 threads for HTTP server
0.00.195.676 I srv         start: binding port with default address family
0.00.197.011 I srv  llama_server: loading model
0.00.197.050 I srv    load_model: loading model '/models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf'
0.01.411.845 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)
0.01.509.829 W srv    load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
0.01.509.863 I common_init_result: fitting params to device memory ...
0.01.509.863 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.03.369.360 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.03.369.994 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.03.404.622 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.05.283.060 W llama_context: n_ctx_seq (190208) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.06.529.690 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.06.638.850 I srv    load_model: loading draft model '/models/mtp-gemma-4-12B-it.gguf'
0.07.088.532 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.07.089.090 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.07.120.677 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.07.332.490 W llama_context: n_ctx_seq (190208) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.07.333.135 W llama_kv_cache: layer   3: sharing with layer 47. k = 0x71147e913040, v = 0x711484bc1040
0.07.333.210 W llama_kv_cache: layer   0: sharing with layer 46. k = 0x71143c21e040, v = 0x71143c65e040
0.07.333.281 W llama_kv_cache: layer   1: sharing with layer 46. k = 0x71143c21e040, v = 0x71143c65e040
0.07.333.283 W llama_kv_cache: layer   2: sharing with layer 46. k = 0x71143c21e040, v = 0x71143c65e040
0.07.677.877 I srv    load_model: initializing slots, n_slots = 1
0.07.800.208 I common_speculative_impl_ngram_mod: adding speculative implementation 'ngram-mod'
0.07.800.229 I common_speculative_impl_ngram_mod: - n_match=24, n_max=64, n_min=48
0.07.800.237 I common_speculative_impl_ngram_mod: - mod size=4194304 (16.000 MB)
0.07.800.253 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.07.800.253 I common_speculative_impl_draft_mtp: - n_max=4, n_min=0, p_min=0.00, n_embd=3840, backend_sampling=1
0.07.800.255 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
0.08.103.907 I srv    load_model: speculative decoding context initialized
0.08.103.928 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 190208
0.08.103.962 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.08.103.962 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.08.103.963 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.08.103.963 I srv    load_model: context checkpoints enabled, max = 1024, min spacing = 256
0.08.103.993 I srv          init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.08.108.960 I init: chat template, example_format: '<|turn>system
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
<|channel>thought
<channel|>'
0.08.109.482 I srv          init: init: chat template, thinking = 0
0.08.109.534 I srv  llama_server: model loaded
0.08.109.551 I srv  llama_server: server is listening on http://0.0.0.0:8400
0.08.109.556 I srv  update_slots: all slots are idle
0.10.214.196 I srv  params_from_: Chat format: peg-gemma4
0.10.214.387 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.10.214.408 I srv  get_availabl: updating prompt cache
0.10.214.416 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.10.214.421 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 190208 tokens, 8589934592 est)
0.10.214.430 I srv  get_availabl: prompt cache update took 0.01 ms
0.10.214.484 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.10.288.209 I slot create_check: id  0 | task 0 | created context checkpoint 1 of 1024 (pos_min = 0, pos_max = 0, n_tokens = 1, size = 0.128 MiB)
0.10.572.520 I slot print_timing: id  0 | task 0 | prompt eval time =     186.88 ms /    15 tokens (   12.46 ms per token,    80.26 tokens per second)
0.10.572.537 I slot print_timing: id  0 | task 0 |        eval time =     170.99 ms /     9 tokens (   19.00 ms per token,    52.63 tokens per second)
0.10.572.537 I slot print_timing: id  0 | task 0 |       total time =     357.88 ms /    24 tokens
0.10.572.544 I slot print_timing: id  0 | task 0 |    graphs reused =          2
0.10.572.544 I slot print_timing: id  0 | task 0 | draft acceptance = 0.87500 (    7 accepted /     8 generated)
0.10.572.568 I statistics        ngram-mod: #calls(b,g,a) =    1      2      0, #gen drafts =      0, #acc drafts =     0, #gen tokens =      0, #acc tokens =     0, dur(b,g,a) = 0.003, 0.002, 0.000 ms
0.10.572.569 I statistics        draft-mtp: #calls(b,g,a) =    1      2      2, #gen drafts =      2, #acc drafts =     2, #gen tokens =      8, #acc tokens =     7, dur(b,g,a) = 0.004, 22.876, 0.006 ms
0.10.572.596 I slot      release: id  0 | task 0 | stop processing: n_tokens = 24, truncated = 0
0.10.572.609 I srv  update_slots: all slots are idle
0.17.647.048 I srv  params_from_: Chat format: peg-gemma4
0.17.647.266 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.306 (> 0.100 thold), f_keep = 0.458
0.17.647.280 I srv  get_availabl: updating prompt cache
0.17.647.481 W srv   prompt_save:  - saving prompt with length 24, total state size = 3.201 MiB (draft: 0.000 MiB)
0.17.648.500 I srv          load:  - looking for better prompt, base f_keep = 0.458, sim = 0.306
0.17.648.517 I srv        update:  - cache state: 1 prompts, 3.329 MiB (limits: 8192.000 MiB, 190208 tokens, 190208 est)
0.17.648.518 I srv        update:    - prompt 0x5bb1744c2d40:      24 tokens, checkpoints:  1,     3.329 MiB
0.17.648.519 I srv  get_availabl: prompt cache update took 1.24 ms
0.17.648.580 I slot launch_slot_: id  0 | task 6 | processing task, is_child = 0
0.17.648.601 I slot update_slots: id  0 | task 6 | Checking checkpoint with [0, 0] against 0...
0.17.648.670 W slot update_slots: id  0 | task 6 | restored context checkpoint (pos_min = 0, pos_max = 0, n_tokens = 1, n_past = 1, size = 0.128 MiB)
0.18.117.467 I begin: ngram_mod occupancy = 12/4194304 (0.00)
0.20.008.537 I slot print_timing: id  0 | task 6 | n_decoded =    103, tg =  54.52 t/s
0.23.021.269 I slot print_timing: id  0 | task 6 | n_decoded =    233, tg =  47.53 t/s
0.26.032.032 I slot print_timing: id  0 | task 6 | n_decoded =    351, tg =  44.36 t/s
0.29.071.137 I slot print_timing: id  0 | task 6 | n_decoded =    463, tg =  42.28 t/s
0.30.267.078 I slot print_timing: id  0 | task 6 | prompt eval time =     470.43 ms /    35 tokens (   13.44 ms per token,    74.40 tokens per second)
0.30.267.098 I slot print_timing: id  0 | task 6 |        eval time =   12147.91 ms /   507 tokens (   23.96 ms per token,    41.74 tokens per second)
0.30.267.099 I slot print_timing: id  0 | task 6 |       total time =   12618.34 ms /   542 tokens
0.30.267.100 I slot print_timing: id  0 | task 6 |    graphs reused =        138
0.30.267.101 I slot print_timing: id  0 | task 6 | draft acceptance = 0.66367 (  369 accepted /   556 generated)
0.30.267.115 I statistics        ngram-mod: #calls(b,g,a) =    2    141      0, #gen drafts =      0, #acc drafts =     0, #gen tokens =      0, #acc tokens =     0, dur(b,g,a) = 0.049, 0.429, 0.000 ms
0.30.267.118 I statistics        draft-mtp: #calls(b,g,a) =    2    141    141, #gen drafts =    141, #acc drafts =   120, #gen tokens =    564, #acc tokens =   376, dur(b,g,a) = 0.007, 1796.018, 0.328 ms
0.30.267.136 I slot      release: id  0 | task 6 | stop processing: n_tokens = 544, truncated = 0
0.30.267.155 I srv  update_slots: all slots are idle
0.41.970.095 I srv  params_from_: Chat format: peg-gemma4
0.41.971.609 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = 1237324259039
0.41.971.623 I srv  get_availabl: updating prompt cache
0.41.971.745 W srv   prompt_save:  - saving prompt with length 544, total state size = 72.529 MiB (draft: 0.000 MiB)
0.42.025.044 I srv          load:  - looking for better prompt, base f_keep = 0.059, sim = 0.057
0.42.025.077 I srv        update:  - cache state: 2 prompts, 75.986 MiB (limits: 8192.000 MiB, 190208 tokens, 190208 est)
0.42.025.078 I srv        update:    - prompt 0x5bb1744c2d40:      24 tokens, checkpoints:  1,     3.329 MiB
0.42.025.079 I srv        update:    - prompt 0x5bb1749792e0:     544 tokens, checkpoints:  1,    72.657 MiB
0.42.025.080 I srv  get_availabl: prompt cache update took 53.46 ms
0.42.025.194 I slot launch_slot_: id  0 | task 149 | processing task, is_child = 0
0.42.025.223 I slot update_slots: id  0 | task 149 | Checking checkpoint with [0, 0] against 0...
0.42.025.364 W slot update_slots: id  0 | task 149 | restored context checkpoint (pos_min = 0, pos_max = 0, n_tokens = 1, n_past = 1, size = 0.128 MiB)
0.44.999.046 I slot create_check: id  0 | task 149 | created context checkpoint 2 of 1024 (pos_min = 0, pos_max = 539, n_tokens = 540, size = 68.562 MiB)
0.45.234.822 I slot print_timing: id  0 | task 149 | prompt processing, n_tokens =    556, progress = 0.99, t =   3.21 s / 173.18 tokens per second
0.45.353.108 I begin: ngram_mod occupancy = 564/4194304 (0.00)
0.49.993.091 I slot print_timing: id  0 | task 149 | n_decoded =    103, tg =  22.21 t/s
0.53.097.285 I slot print_timing: id  0 | task 149 | n_decoded =    162, tg =  20.93 t/s
0.56.138.165 I slot print_timing: id  0 | task 149 | n_decoded =    225, tg =  20.87 t/s
0.59.268.134 I slot print_timing: id  0 | task 149 | n_decoded =    281, tg =  20.20 t/s
1.02.278.709 I slot print_timing: id  0 | task 149 | n_decoded =    330, tg =  19.50 t/s
1.05.402.305 I slot print_timing: id  0 | task 149 | n_decoded =    381, tg =  19.01 t/s
1.08.513.946 I slot print_timing: id  0 | task 149 | n_decoded =    422, tg =  18.22 t/s
1.11.558.671 I slot print_timing: id  0 | task 149 | n_decoded =    472, tg =  18.01 t/s
1.14.677.438 I slot print_timing: id  0 | task 149 | n_decoded =    519, tg =  17.70 t/s
1.17.734.849 I slot print_timing: id  0 | task 149 | n_decoded =    572, tg =  17.66 t/s
1.20.884.097 I slot print_timing: id  0 | task 149 | n_decoded =    623, tg =  17.53 t/s
1.23.920.269 I slot print_timing: id  0 | task 149 | n_decoded =    671, tg =  17.40 t/s
1.26.925.612 I slot print_timing: id  0 | task 149 | n_decoded =    725, tg =  17.44 t/s
1.29.943.629 I slot print_timing: id  0 | task 149 | n_decoded =    762, tg =  17.09 t/s
1.32.985.324 I slot print_timing: id  0 | task 149 | n_decoded =    800, tg =  16.80 t/s
1.33.373.387 I slot print_timing: id  0 | task 149 | prompt eval time =    3331.18 ms /   560 tokens (    5.95 ms per token,   168.11 tokens per second)
1.33.373.404 I slot print_timing: id  0 | task 149 |        eval time =   48019.06 ms /   805 tokens (   59.65 ms per token,    16.76 tokens per second)
1.33.373.405 I slot print_timing: id  0 | task 149 |       total time =   51350.25 ms /  1365 tokens
1.33.373.406 I slot print_timing: id  0 | task 149 |    graphs reused =        429
1.33.373.407 I slot print_timing: id  0 | task 149 | draft acceptance = 0.43136 (  509 accepted /  1180 generated)
1.33.373.421 I statistics        ngram-mod: #calls(b,g,a) =    3    436      0, #gen drafts =      0, #acc drafts =     0, #gen tokens =      0, #acc tokens =     0, dur(b,g,a) = 0.145, 1.261, 0.000 ms
1.33.373.423 I statistics        draft-mtp: #calls(b,g,a) =    3    436    436, #gen drafts =    436, #acc drafts =   331, #gen tokens =   1744, #acc tokens =   885, dur(b,g,a) = 0.010, 7329.998, 0.926 ms
1.33.373.543 I slot      release: id  0 | task 149 | stop processing: n_tokens = 1365, truncated = 0
1.33.373.579 I srv  update_slots: all slots are idle
  4. github-actions
    added
    stale
    on Jul 15, 2026

  5. github-actions commented on Jul 29, 2026

    @github-actions
    github-actionsBot
    on Jul 29, 2026 – with GitHub Actions
    Contributor

    This issue was closed because it has been inactive for 14 days since being marked as stale.

  6. github-actions
    closed this as not plannedon Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug-unconfirmedstale

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions