Bundled draft_vocab.bin has almost no CJK tokens: MTP acceptance collapses on Chinese output (+14% decode without it)

Hi Niko, following up on #71 with measurements from a full run of Strata on our box. Everything below is from one machine, a single user, 2026-09-29.

Setup

RTX 5070 Ti 16 GB (PCIe 5.0 x16, your probe reports 49.8 GB/s), Ryzen 7 7700 (8 cores), 2×48 GB DDR5-5200, Ubuntu 24.04, CUDA 13.2
Engine 0.1.24 (3ce2523), built manually with cmake for sm_120
ISTA-DASLab GSQ-RCO IQ3_XXS, MTP packed q2_0 via mtp_fetch/mtp_pack/mtp_rt, data/draft_vocab.bin copied into rt/ (as setup.py does)
Flags: --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --max-context 131072 --kv int8 --kv-resident 32768 --vision (CPU encoder)

Finding

data/draft_vocab.bin holds 40,525 token ids. When I decode them with the model's tokenizer, only 27 of those tokens contain a CJK character. The full vocabulary has 55,328 such tokens. As a result, the MTP head can almost never propose a Chinese token, and draft acceptance collapses whenever the answer is in Chinese.

To test this, I pointed --mtp at a copy of rt/ that leaves out draft_vocab.bin, so the draft uses the full native head. Nothing else changed. The benchmark was 3 fixed prompts × 2 rounds at temperature 0 with thinking off, and the numbers come from /metrics plus the engine log:

prompt (output language)	bundled draft_vocab	no draft_vocab
describe a 6K-token table (Chinese)	65–67 tok/s, drafts 56/121	78 tok/s, drafts 127/188
explain a concept, ~300 chars (Chinese)	70–73 tok/s, drafts 42/131	84–87 tok/s, drafts 97/143
write an LRU cache (Python + Chinese notes)	90–95 tok/s	99–100 tok/s
average	76.8	87.7 (+14%)

VRAM was not worse. The log shows 336 MiB free with everything loaded, against 266 MiB with the subset head.
The outputs look normal. I spot-checked all three answers.

Suggestion (your call)

Build the draft subset from a multilingual corpus that includes CJK.
Or add a flag/setting to use the full head, and mention it in the README for non-English users.

Other data points from the same box (FYI, no action needed)

--pcie-frac: with the 0.55 default my average was 76.8; 0.8 gave 72.7 and 1.0 gave 67.7. So even on PCIe 5.0 x16, the 0.55 cap looks right.
--spec 2 gave about the same result as --spec 4 (76.2 vs 76.8 with the subset head; 85.8 vs 87.7 with the full head).
I also tried experts-per-token = 8, with a local patch: an env override in generate.cpp, plus relaxing the k == 10 checks in prefill.cpp/verify.cpp, where prefill stays at 10. Decode did not improve: 76.7 vs 76.8. On a llama.cpp-based fork the same change gives us +10–18%, so on Strata the expert path clearly isn't the bottleneck.
Against our tuned llama.cpp fork on the same box and model class:
- Prefill is 2,590–2,830 vs 660–750 tok/s, about 3.9×.
- At 128K, decode is 79.7 vs 81.3.
- Multi-turn prefix reuse works (19.6K reused, 0.2–0.5 s).
- Vision matches.
- A 24-question domain exam scored 23/24 on both.

Thanks for the fast iteration. Happy to rerun anything on this hardware.

Draft reply for #137 — local reproduction on Windows + RTX 5070 Ti 16 GB

Not posted. Paste-ready as-is; the two tools it references (tools/draft_vocab.py, tools/mtp_cjk_bench.py) are
written locally and can be attached or pushed to a branch on request.

Confirmed on a 16 GB card, and the effect is larger than the +14% in the title once the workload is actually
Chinese: +45% decode on a real translation job, and the fix costs no expert cache at all if you expand the
subset instead of dropping it.

Setup. Windows 11 x64, RTX 5070 Ti (16,303 MiB), engine 0.1.24, pack qwen3.8-flash-next-iq2_xs-vision-cpu,
--expert-cache auto, temperature=0, thinking disabled, 2 rounds per prompt, one engine restart per head
configuration (the head is chosen when the draft layer binds). All tok/s, token counts and draft figures below are
the engine's own numbers read from the DONE line of the log, not wall-clock estimates.

1. The shipped file is exactly as reported

data/draft_vocab.bin on this install is 162,100 bytes = 40,525 int32 ids, and the engine says so at bind time:

strata mtp: draft head over 40525 tokens (52.6 MiB)

Counting scripts over the decoded token (byte-level spellings decoded first, so this is per character class):

set	ids	Han	kana	Hangul	CJK punct	Latin
shipped subset	40,525	27	70	36	35	37,710
full vocabulary	248,320	55,328	5,455	6,793	252	—

27 Han tokens out of 55,328. What that means for a Chinese answer, measured with the real BPE tokenizer over the
10,939 characters the model actually produced in the translation job below (6,841 token occurrences):

head	occurrences it could propose	Han occurrences	Han kinds
shipped subset	1,618 / 6,841 (23.7%)	0 / 5,101 (0.0%)	0 / 1,254
English subset ∪ all CJK	6,833 / 6,841 (99.8%)	5,101 / 5,101 (100%)	1,254 / 1,254

A greedy draft can only propose a token the head can emit, so on Chinese output the shipped head drafts nothing at
all; every Han token costs a full target-model step.

2. Three workloads, three heads

Measured per configuration: shipped subset (52.6 MiB), English subset ∪ all CJK (137.9 MiB), and no draft_vocab.bin
at all, i.e. the full native Q5_K head (322.1 MiB, 337,715,200 bytes).

a. Real job — English novel → Chinese translation. 8 requests, 6,837 generated tokens.

head	mean tok/s	accept	wall
shipped subset	68.1	34–36%	108.9 s
CJK-expanded subset	99.0	70–77%	78.5 s
full native head	96.4	70–77%	80.2 s

+45.4% mean tok/s, −27.9% wall clock. The translations themselves agree 97.8–99.6% with the baseline run and the
glossary held across segments, so this is not a quality trade.

b. Synthetic suite (4 fixed prompts: a 6K-token Chinese table, a Chinese concept explanation, Chinese code,
English code).

head	Chinese prompts	English code
shipped subset	71.6 (24–63% accept)	101.2
CJK-expanded subset	88.9 (58–82%)	99.8
full native head	88.5 (57–83%)	100.4

English is unaffected either way; the gap is entirely on the Chinese side, and it collapses where the shipped subset
has nothing to draft (concept-zh: 24% accept, 67.5 tok/s → 91.0).

c. Coding task — a four-operations calculator written in answer to Chinese instructions (4 steps × 2 rounds).
Output here is code with Chinese comments, so Han is only 3–17% of the generated tokens, and the margin tracks that
share step by step:

step	Han share of output	shipped	CJK-expanded	margin
tokenize (comment-heavy)	12–17%	71.7	83.9	+17.0%
parser	3–4%	92.0	99.1	+7.7%
evaluator + asserts	4–7%	94.8	95.6	+0.8%
argparse CLI	6–9%	92.2	97.2	+5.4%

Overall 87.7 → 93.9 tok/s (+7.1%). This is the useful negative control: the head only earns what the answer's Han
fraction pays for it, so the issue is specifically about CJK output, not about Chinese prompts.

3. What I would ship, and why not the full head

The expanded subset is one command (tools/draft_vocab.py is referenced by src/core/mtp.cpp:301 but is not in the
repo — I wrote it to build and audit these):

python tools/draft_vocab.py --tokenizer <pack>/tokenizer --base draft_vocab.bin \
    --add cjk --row-bytes 1360 --out draft_vocab_cjk.bin
# 106285 ids: han=55328 kana=5455 hangul=6793 cjk_punct=252 latin=37711 -> head 137.9 MiB

On this card the extra 85 MiB comes out of the reserve rather than the expert cache:

head	head size	`expert cache N slots`
shipped subset	52.6 MiB	5627
CJK-expanded subset	137.9 MiB	5627 (unchanged)
full native head	322.1 MiB	5551 (−76 slots)

So expanding the subset buys the whole margin for no cache, while removing the file costs 76 expert slots and is not
even faster: the full head measured slightly below the expanded subset on both Chinese jobs (96.4 vs 99.0 tok/s on
the translation, 92.5 vs 93.9 on the coding job), with acceptance the same on translation and 3 points higher on the
coding job. If the goal is a single portable default, --add cjk on top of the existing English subset is the
better trade; and the same tool exposes han / kana / hangul / cjk_punct separately for packs that need to
stay small (Han alone is 55,328 rows).

Two honest caveats: on mostly-Latin output the expanded head measured 1.4% below the shipped one (99.8 vs 101.2),
which is at noise level but is the only cost I saw; and I cannot claim any effect on code quality from these runs —
at temperature=0 the same head's two rounds agree only 73.2% of the time at the median, while different heads on the
same round agree 75.9%, so the model's own spread swamps anything the head might do.

Harness for reproducing all of the above: python tools/mtp_cjk_bench.py --label <head> [--translate <file.txt> |--job <tasks.txt>] --rounds 2 --keep-answers --out <dir>, one restart between labels; --parse-log and --dry-run
work offline.

@demon851113

Thanks @demon851113 and @qwased, confirmed: the bundled subset has 27 Han tokens against 55,328 in the vocabulary. We'll ship a subset that includes CJK (your measurements show it costs no expert slots on 16 GB) and have setup refresh it on existing installs. @qwased, a PR with tools/draft_vocab.py would be very welcome.

tools/draft_vocab.py: the draft head's token subset, with whole scrip…

data/draft_vocab.bin: the draft subset with all Chinese, Japanese and…

@demon851113

Fixed in 0.1.27: the draft subset now has every Chinese, Japanese and Korean token (40,525 -> 106,299 ids), and setup replaces the old subset on existing installs at the next start (a subset you made yourself is kept). On an RTX 5070 with Q2_0, answers in Chinese are 15-38% faster (a concept explanation 55 -> 76 tokens/s, a translation 67 -> 81) and Japanese 63 -> 72. English costs 1-2% (the larger draft head), and the number of cached experts stays the same on 12 GB. tools/draft_vocab.py builds and inspects subsets (--add han,kana,hangul,cjk_punct or cjk, --stats). Thanks @demon851113 and @qwased for the clear measurements!

setup: --host/--api-key on a start are kept for the model (Niko1221#179…

tools/draft_vocab.py: the draft head's token subset, with whole scrip…

data/draft_vocab.bin: the draft subset with all Chinese, Japanese and…

tools/draft_vocab.py: the draft head's token subset, with whole scrip…

data/draft_vocab.bin: the draft subset with all Chinese, Japanese and…

Navigation Menu

Repository navigation

Bundled draft_vocab.bin has almost no CJK tokens: MTP acceptance collapses on Chinese output (+14% decode without it) #137

Draft reply for #137 — local reproduction on Windows + RTX 5070 Ti 16 GB

1. The shipped file is exactly as reported

2. Three workloads, three heads

3. What I would ship, and why not the full head

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Bundled draft_vocab.bin has almost no CJK tokens: MTP acceptance collapses on Chinese output (+14% decode without it) #137

Description

Activity

qwased commented on Sep 29, 2026

Draft reply for #137 — local reproduction on Windows + RTX 5070 Ti 16 GB

1. The shipped file is exactly as reported

2. Three workloads, three heads

3. What I would ship, and why not the full head

Niko1221 commented on Sep 29, 2026

Niko1221 commented on Sep 29, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions