[tkran2]

Overview

The problem was that llama_detokenize deletes the space before ! ? . , and prior to 's 'm 're 've for certain models. As an example, Gemma 4 converts x != y into x!= y, while the text is left unchanged by Hugging Face. This happened because llama.cpp, for BPE vocabs, by default sets clean up on. The Hugging Face setting (clean_up_tokenization_spaces), which is either not set or false for these specific models, was never stored by the GGUF. My PR does two things: First, the converter saves the setting into a new GGUF key, tokenizer.ggml.clean_spaces, and the loader then reads it. Also, clean up is now by default turned off for the pre-tokenizers that are affected (gemma4, lfm2, glm4/glm5, chatglm-bpe, exaone4, mellum2, minicpm5, dbrx), which means that already existing GGUFs are also fixed.

Fixes #30905

Additional information

Before the fix, the --check-decode test did not pass and had 2 decode errors. After the fix it passes. All 15 test-tokenizer-0 tests pass, and InferCheck decode disagreements decreased to 0 for GLM-4, GLM-4.5, GLM-4.7, LFM2, MiniCPM5, EXAONE 4 and Phi-4.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. I used Claude to help write the code and check Hugging Face configs to see which models were affected. I also used Claude to write the --check-decode test. I reviewed this code and then ran the tests myself. I also wrote the PR and commit message myself.

[CISC]

@ggerganov This is generally merge-ready I'd just like a second review. :)

Also wondering if it makes sense to run --check-decode on more/all vocabs (including ggml-org/vocabs)?

[ggerganov]

Also wondering if it makes sense to run --check-decode on more/all vocabs (including ggml-org/vocabs)?

We can do that, but maybe after we move all tokenizer tests in a dedicated workflow? Currently they run through ctest and execute for a lot of jobs - not needed.