Repository navigation
Misc. bug: llama-quantize fails on DeepSeek-V4 — i32 ffn_gate_tid2eid routing table not excluded from quantization #25754
Description
Summary
llama-quantize fails on DeepSeek-V4 models because the arch's ffn_gate_tid2eid tensor is i32 and isn't excluded from quantization.
Repro
Convert any DeepSeek-V4 checkpoint (e.g. deepseek-ai/DeepSeek-V4-Flash-DSpark) and quantize it:
python convert_hf_to_gguf.py DeepSeek-V4-Flash-DSpark --outfile v4-q8.gguf --outtype q8_0 --fp8-as-q8
llama-quantize --allow-requantize v4-q8.gguf v4-q4.gguf Q4_K_MOutput:
[ 20/1328] blk.0.ffn_gate_shexp.weight - [4096, 2048, 1, 1], type = q8_0, converting to q4_K .. size = 8.50 MiB -> 4.50 MiB
[ 21/1328] blk.0.ffn_gate_tid2eid.weight - [ 6, 129280, 1, 1], type = i32,
llama_model_quantize: failed to quantize: cannot dequantize/convert tensor type i32
(thrown at src/llama-quant.cpp:228)
Cause
blk.N.ffn_gate_tid2eid.weight (LLM_TENSOR_FFN_GATE_TID2EID, declared in src/llama-arch.cpp and created in src/models/deepseek4.cpp as {n_expert_used, n_vocab}) is a token-id → expert-id index table, not weights. It's i32, so llama_tensor_quantize_impl can't convert it.
llama_tensor_get_type() in src/llama-quant.cpp already excludes the other MoE routing tensor:
quantize &= name.find("ffn_gate_inp.weight") == std::string::npos;but tid2eid was never added when DeepSeek-V4 landed (#24162) — understandable, since that work targeted inference.
Suggested fix
One line, next to the existing router exclusion:
quantize &= name.find("ffn_gate_inp.weight") == std::string::npos;
quantize &= name.find("tid2eid") == std::string::npos; // DeepSeek-V4: i32 token-id -> expert-id tableWith this, the tensor is copied through verbatim (2.959 MiB) and quantization completes.
I'm happy to open a PR if that's useful — just say the word (or feel free to take the line directly, it's trivial).
Notes
- Probably went unnoticed because the public V4 releases already ship quantized (FP8 attention + MXFP4 experts, ~4.4 BPW), so there's little reason to run
llama-quantizeon them. It does matter if you want to push the routed experts below 4 bits. - Tested on
505b1ed1, macOS/Metal.
Thanks for the V4 support — it landed fast and the conversion path (incl. --fp8-as-q8) worked flawlessly otherwise.
Activity
fairydreaming commentedon Jul 17, 2026
@KikoCis Just a friendly reminder that your quantization command will likely make the model significantly dumber. The problem is that llama-quantize will try to quantize most of the model tensors.
If you check for example unsloth DeepSeek V4 quants they have almost all the model tensors untouched (kept in high-resolution f32, f16 or Q8_0), only huge expert tensors that take most of the disk space are quantized more.
KikoCis commentedon Jul 18, 2026
Thanks @fairydreaming — you're right, and I appreciate the nudge. A flat quantize crushes the small, sensitive tensors (attention, embeddings, norms) for almost no disk saving on a MoE, where the experts are the real weight. Keeping those high and only pushing the expert tensors down (à la Unsloth's UD /
--tensor-type) is clearly the way.For context, I wasn't actually shipping that quant — I was requantizing from a community Q8_0 just to test, and it turned out pointless regardless: its experts are already mxfp4, so a flat Q4_K up-converts them and the file grows. The tid2eid report was just to stop the quantize from erroring out; good to see #25787 handling it. Thanks again for taking the time.
llama-quant : exclude i32 ffn_gate_tid2eid routing table from quantiz…
Sync master with upstream ggml-org/llama.cpp (82bb485) (#80)
llama-quant : exclude i32 ffn_gate_tid2eid routing table from quantiz…
llama-quant : exclude i32 ffn_gate_tid2eid routing table from quantiz…