Repository navigation
Misc. bug: llama_detokenize removes the space preceding "!", "?", ".", ",", and prior to "'s", "'re" for Gemma 4 and other vocabs whose Hugging Face tokenizer does not, so x != y turns into x!= y #30905
Description
Name and Version
b11769-14-g339f3242c
Operating systems
Mac
Which llama.cpp modules do you know to be affected?
libllama (core library)
Command line
# harness: https://github.com/tkran2/infercheck/blob/main/harness/llama_tok/llamatok.cpp (llama_tokenize + llama_detokenize)
python - <<'EOF'
import subprocess
V='../llama.cpp/models/ggml-vocab-gemma-4.gguf'; T='harness/llama_tok/llamatok'
for s in ['if (x != y) {', 'x = a ? b : c', "It 's fine"]:
ids=subprocess.run([T,V],input=s.encode().hex()+'\n',capture_output=True,text=True).stdout.split('\t')[0]
dec=subprocess.run([T,V,'--decode'],input=ids+'\n',capture_output=True,text=True).stdout.split('\n')[0]
print(repr(s), '->', repr(bytes.fromhex(dec).decode()))
EOF
'if (x != y) {' -> 'if (x!= y) {'
'x = a ? b : c' -> 'x = a? b : c'
"It 's fine" -> "It's fine"
python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('google/gemma-4-12b-it'); ids=t.encode('if (x != y) {', add_special_tokens=False); print(ids, repr(t.decode(ids)))"
[584, 568, 236781, 2843, 570, 236768, 642] 'if (x != y) {'Problem description & steps to reproduce
The spaces before . , ! ? and before 's 'm 're 've are deleted by common_detokenize as well as llama_detokenize. The text is unchanged by Hugging Face; clean_up_tokenization_spaces is not set or it is false for these models. The cause of this problem is that clean_spaces = true is set in llama-vocab.cpp for every one of the BPE vocabs and it is turned off for just some of the pretokenizers (qwen2, deepseek, qwen35, and command-r). The Hugging Face setting is not stored by the GGUF. The models which are affected are Gemma 4, GLM-4.x/5.x, LFM2, Mellum2, Nemotron (pixtral), Granite 4.1 (dbrx), and MiniCPM5. This can be seen by users in llama-mtmd-cli chat history, which is afterwards retokenized, the server's prompt field, and any app which calls llama_detokenize. Streamed output on the other hand is not affected. The scale of this issue is that it affects 71 of the 500 most downloaded GGUF repos, with 20.7 million downloads in only 30 days. A related PR is #28472, which just adds a way of reading the setting. A question I have: store the Hugging Face's setting in the GGUF or change the default? I would be happy to open the PR.
First Bad Commit
Not a regression: the clean-up pass was added in #8039 (July 2024).
Activity
stariver2000 commentedon Oct 21, 2026
Hi, I reproduced it on Linux (master
0a1ef309c).In the gemma4 branch, if I add
clean_spaces = false;, it is fixed.test-tokenizer-0also passes.pre_type = LLAMA_VOCAB_PRE_TYPE_GEMMA4; escape_whitespaces = true; + clean_spaces = false;Do you have a plan to open a PR? Or can I open a PR for this?
nikwen commentedon Oct 21, 2026
For another property, #30882 both changed the conversion script and changed the default for the affected model.
Maybe it's worth doing something similar here for the models you listed?
tkran2 commentedon Oct 21, 2026
Thank you @stariver2000 for the Linux repro on
0a1ef309cand for verifying thattest-tokenizer-0still passes withclean_spaces = falsein the gemma4 branch. I will be opening the PR, but thank you for offering, though I would appreciate it if you wanted to test it.The plan:
clean_up_tokenization_spacesis set bytokenizer_config.json, it will be written to a new key,tokenizer.ggml.clean_spaces.add_space_prefixis read. If it's not there, keep the per-pre-tokenizer default.gemma4,lfm2,glm4/glm5,mellum2,minicpm5, as well asdbrx.lfm2changes by itself. The other tags from the llama3 group remain as they are.Evidence: every model in the issue has set
clean_up_tokenization_spacesto false or does not have it set. Unset in Gemma 4 and DFM-Mimir and false in GLM-4, 4.5, 4.7, and 5.x; LFM2.5; Mellum2; MiniCPM5; Granite 4.1; Phi-4; granite-embedding-311m.databricks/dbrx-base, which is where thedbrxhash came from, does not exist on Hugging Face any longer. Thus, Granite 4.1 and Phi-4 are left as the GGUFs that use this tag.