Repository navigation
Misc. bug: Qwen3.8-Flash-Next (qwen4_exp): 51B N-gram embedding table not loaded after build 10665, causing extremely slow prefill #28355
Description
Name and Version
10665
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
“Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --host 188.0.0.1 --port 5000 -c 262144 -ngl 99 -t 16 -b 2048 -ub 512 -np 2 -mg 2 --tensor-split 0.4,1.6 --no-mmap --temp 1.00 --top-k 20 --min-p 0.00 --repeat-penalty 1.00 -n 131072 --chat-template-kwargs {"reasoning_effort":"medium"} --cache-type-k f16 --cache-type-v f16 -ot per_layer_token_embd=CPU --jinja "Problem description & steps to reproduce
The model should load completely, including the ~51B-parameter N-gram embedding table (20M bigram+trigram entries, located near layer 2), and prefill should run at normal speed — as it does with community build 10665.
Current Behavior
With community prebuilt binaries newer than build 10665, the model loads and can generate output, but the N-gram embedding table is not loaded into memory (not resident). Prefill becomes extremely slow as a result.
Hypothesis (not confirmed): the newer builds fail to load/pin the N-gram embedding table, and the per-token table lookups then hit host memory / mmap page faults instead of being prefetched/overlapped, which destroys prefill performance. Generation output content is still correct.
“Qwen3.8-Flash-Next-UD-Q4_K_L.gguf，dual RTX PRO 5000 72G， 160G DDR5“
First Bad Commit
No response
Activity
ServeurpersoCom commentedon Sep 4, 2026
Hello,
Not a loading failure, a new default. Since #27794 the loader can read large per-layer embedding tensors on demand, and #27969 made
--lazy-mode autothe default: lazy above 4 GiB. The 27 GiB n-gram table (per_layer_token_embd) is above that, so it is read from disk row by row at prefill instead of sitting in RAM.--no-mmapdoes not change this, a lazy tensor forces the file to be mapped anyway (llama-model-loader.cpp), and a mapped table only lives in the page cache, so other disk activity or loading another model evicts it and the next prefill goes back to the SSD.Force the table resident with
--lazy-mode off -ot per_layer_token_embd=CPU(-lzm offis the short form). The-otputs it in system RAM rather than VRAM, with 160 GB of DDR5 it fits easily.For reference, my production runs llama-server in router mode (
--models-preset backend.ini), so the same settings live in an ini file. The[*]section applies to every model, then the real section I run for this model:[*]
no-mmap = on ; Disable mmap (default: off)
mlock = on ; Lock model in RAM
fit = off ; Disable automatic memory fitting (default: on)
ngl = 999 ; Full GPU offload
ctk = q8_0 ; KV cache key quantization
ctv = q8_0 ; KV cache value quantization
fa = on ; Enable flash attention
np = 4 ; Parallel request batching (default: 4)
kvu = on ; Unified KV cache buffer (default: on)
b = 4096 ; Logical maximum batch size (default: 2048)
ub = 2048 ; Physical maximum batch size (default: 512)
[MoE-Vision-Qwen3.8-Flash-Next-125B-A6B]
m = unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
mm = unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-BF16.gguf
reasoning-effort = medium
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0
presence-penalty = 0
repeat-penalty = 1
c = 262144
; Performance critical: without these two, the 27 GiB n-gram table stays mmap'd on the SSD
; and prefill collapses from 2400 to 1000 t/s once another model has evicted the page cache
lazy-mode = off ; Read the n-gram table up front (default: auto, lazy above 4 GiB)
ot = per_layer_token_embd=CPU ; Required with lazy-mode off, 27 GiB does not fit in VRAM
--lazy-mode onandautoremain useful on machines without the RAM for the table, they trade prefill speed for footprint.ServeurpersoCom commentedon Sep 4, 2026
Putting the n-gram table in VRAM would be a waste: it is a per-token row lookup, not a matmul, and the prefill/decode difference between VRAM and system RAM for it is negligible, so keep those 27 GiB of VRAM for context.
checkpoint51677 commentedon Sep 4, 2026
Thank you~ Ok I know the reason now~
fix(llm): roll back to Qwen3.6-35B-A3B, flash-next needs a newer llam…