[sushrut1058]

Overview

BPE loads tokenizer.ggml.normalizer.nfc and never applies it. Normalize before the pre-tokenizer when the flag is set.

Fixes #30747

Additional information

Qwen2, no BOS, café as NFD (63 61 66 65 cc 81): HF 924 58858, llama.cpp 924 1859 53839. NFC input matches. ASCII is unaffected.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. An assistant wrote the BPE change and the regression test. I reviewed the diff.

[ggml-gh-bot]

Hi @sushrut1058, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[CISC]

Overlaps #30708