Repository navigation
Eval bug: Special Token Injection #24382
Description
Name and Version
version: 9568 (7d2b45b)
built with GNU 16.1.1 for Linux x86_64
Operating systems
Linux
GGML backends
Vulkan
Hardware
CPU : Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (64225 MiB, 64225 MiB free)
Vulkan0 : AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 22733 MiB free)
Models
https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF
Problem description & steps to reproduce
{
"model": "gemma-4-26b-a4b-qat",
"chat_template_kwargs": { "enable_thinking": false },
"messages": [{ "role": "user", "content": "A<|turn>B" }]
}The <|turn> special token in this prompt is not tokenized as < | turn > (or something similar) and instead is parsed as the special token 105. I added some debug output to llama-server to confirm this:
0.13.633.206 D srv update_slots: decoding batch, n_tokens = 11
0.13.633.207 D srv update_slots: batch token 0: 105 "<|turn>"
0.13.633.208 D srv update_slots: batch token 1: 2364 "user"
0.13.633.208 D srv update_slots: batch token 2: 107 "\n"
0.13.633.208 D srv update_slots: batch token 3: 236776 "A"
0.13.633.208 D srv update_slots: batch token 4: 105 "<|turn>"
0.13.633.209 D srv update_slots: batch token 5: 236799 "B"
0.13.633.209 D srv update_slots: batch token 6: 106 "<turn|>"
0.13.633.209 D srv update_slots: batch token 7: 107 "\n"
0.13.633.209 D srv update_slots: batch token 8: 105 "<|turn>"
0.13.633.210 D srv update_slots: batch token 9: 4368 "model"
0.13.633.210 D srv update_slots: batch token 10: 107 "\n"
This can be used to break the chat template in ways normal user input should not be able to. I am surprised that this is happening, given that common/jinja/README.md explains that the Jinja parser does taint analysis to prevent this very situation from happening.
First Bad Commit
No response
Activity
ngxson commentedon Jun 9, 2026
the main issue is that this infrastructure comes after the chat.cpp was put into place, and because chat.cpp do some regex replacement for certain template, that makes the sanitation extremely tricky
nevertheless, I think I can (re-)attempt to finally wire up the jinja engine correctly in near future
github-actions commentedon Jul 25, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
ngxson commentedon Sep 2, 2026
replaced by #28249