Repository navigation
CJK output is systematically wrong at IQ1_M: 0/10 correct on Chinese, 10/10 on English (greedy) #438
Description
Affected: Engine 0.1.30 (2a57d8d), native i-quant expert packs, HIP/gfx1100 build
(ROCm 10.0). Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M (the Coder GGUF, IQ1_M = 1-bit).
Reported against the Coder repo's only published quant.
At temperature 0, Chinese output is never correct on this box. It either degrades into a
single-token loop or drops a character. English is unaffected. This is reproducible on every
run and is not a sampling artifact.
Reproduction
Server: serve.server --engine strata with the standard Coder config
(--expert-cache auto --mmap-experts --mtp rt --spec 4 --spec-min-p 0.5 --prefill auto --max-context 65536 --kv int8 --vision), reasoning off.
Prompt: 用一句话说明你是哪个模型，跑在什么硬件上。
Two distinct failure modes, both wrong:
我是通（通通通通通通通通通通通通通通… finish_reason=length ← degenerate, single-token loop
我是通千（Qwen），由阿里巴巴通千团队开发 ← no loop, but 「通义千问」 lost the 「义」
The second one is the more informative case: the answer is structurally correct and the
model clearly wants to say 通义千问, but it emits 通千. The character 义 is not merely
misordered — it is absent.
Quantified
Same prompt, temperature: 0, reasoning_effort: none, n=10 per language:
| language | correct | wrong-but-fluent | degenerate loop |
|---|---|---|---|
| English (equivalent question) | 10 | 0 | 0 |
| Chinese | 0 | 5 | 5 |
English answers are byte-identical across all 10 runs. Chinese is 0-for-10 correct.
What I ruled out (each by ablation, not by reasoning)
| variable | experiment | result |
|---|---|---|
| suffix (prompt-lookup) drafter | --suffix-draft 0 |
still degenerates |
| verify window width | --spec 2 vs 4 |
still degenerates |
| speculative accept path | engine reports 63/71 and 48/48 drafts accepted | accept logic is correct — it only commits tokens the target verified (generate.cpp:4761) |
| GPU vs CPU expert rounding | --expert-cache 1500 → 9019 slots (hit 62% → 97%) |
degenerates at every size |
| KV quantization | --kv fp16, --kv k8v4, full-VRAM config |
still degenerates |
| sampling | temperature: 0 (greedy) |
still degenerates |
| repetition penalties | penalty_repeat 1.15/1.2, penalty_freq, penalty_present, with penalty_last_n 64/256 |
no effect; does not escape even at 300 tokens |
| draft-independence switch | STRATA_IQ_MT_MIN=1 (which 0.1.30 setup now requires, #152) |
5/5 still degenerate |
| run-to-run variance | 6 consecutive identical greedy requests | 6/6 degenerate (stable) |
A caution about single-sample conclusions: one --kv fp16 run came back clean and looked
like the root cause. Re-running that exact config gave 3 clean / 1 degenerate over 4 runs, then
5 wrong / 5 degenerate over 10. The clean run was luck. fp16 only shifts the failure mix; it
fixes nothing.
Observations that may be relevant
- The failure is CJK-specific. English is unaffected, which argues against a general
numerical problem. - The pack's
vocab.jsoncontains no merged CJK tokens at all —通,千,义are not
keys in the 248,320-entry vocab, and zero entries contain a CJK character. There are 66,744
byte-fallback entries. CJK is byte-mapped, consistent with [tokenizer] O(n^2) BPE merge loop: an unbroken CJK run is ~10 s per 8K chars (1266x slower than needed) #268's finding that the
tokenizer byte-maps each CJK character to 1–3 byte-chars. - IQ1_M is 1-bit (
vec_dot_iq1_m_q8_1,src/kernels/cuda/iq_kernels.cu:223). Combined
with (2), a Han character costs 3 tokens of a 1-bit budget, versus 1 token for an English
word — plausibly a large effective precision deficit exactly where the failures land. draft_vocab.binis absent frommtp/rt/on this install (onlydense.bin,
dense.txt,experts.bin), and this build's--helphas no--draft-vocaboption, so the
CJK draft subset described indocs/DETAILS.md:82is not present here. Per Bundled draft_vocab.bin has almost no CJK tokens: MTP acceptance collapses on Chinese output (+14% decode without it) #137 a missing
draft_vocab.binmeans the full native head, which was measured as faster, so I do not
think this causes the quality failure — but it means the documented CJK drafting path does
not exist in this build.
Why this was not caught by the test suite
grep -rlP '[\x{4e00}-\x{9fff}]' bench/ tests/ returns nothing — no file under bench/ or
tests/ contains a Chinese character. Every quality result in the repo rests on English or
synthetic prompts.
Related: bench/results/2026-09-27-kv-q4/README.md concludes "int8 is indistinguishable from
fp16" from mean KL and perplexity on English code/document/chat text. A mean-based metric
cannot catch this class of failure: the flip happens at one near-tied token, the mean barely
moves, but greedy decoding locks in permanently.
Questions
- Is a 1-bit quant expected to support CJK at all, or is IQ1_M intended as English/code-only?
docs/DETAILS.mdadvertises CJK draft-vocab support for 15–38% faster CJK answers, which
implies CJK is meant to work. - Is there a known-good (non-1-bit) configuration on this model that produces correct Chinese?
TheCoderHF repo publishes only IQ1_M; IQ3_XXS / Q2_0 exist in the separate
…-GSQ-RCO-GGUFrepo, which is a different fine-tune, so I could not run a clean
quant-only ablation. - Would a CJK prompt set in
bench/(even a small one) be accepted? It looks like the gap
that let this through.
Environment
- RX 7900 XTX (gfx1100), ROCm 10.0, engine built locally with hipcc
- Xeon E5-2696 v4 (AVX2, no AVX-512), 62 GB RAM
- Engine 0.1.30, commit
2a57d8d
Related / not the same as
#137 (missing draft_vocab.bin collapses CJK draft acceptance — a speed issue; a missing file means
the full native head, measured faster there) and #268 (tokenizer byte-maps CJK 3x, consistent with (2)
above). #152 is closed and covers CPU expert arithmetic varying with verifier width: with
STRATA_IQ_MT_MIN=1 on, and at --spec 2, and across every expert-cache size, CJK output is still wrong
5/5 — so I do not think #152 is the cause.
Activity
lybhb8 commentedon Oct 2, 2026
Correction: IQ1_M is not a 1-bit format — retracting my bit-width argument
@lybhb8 — I got a detail wrong in my original report and want to correct it before it misleads
anyone.
I wrote that "IQ1_M is 1-bit" and argued from it that a Han character costs 3 tokens of a 1-bit
budget versus 1 token for an English word. That argument does not hold, and I'm withdrawing it.
docs/DETAILS.md:169states it precisely: IQ1_M is named for 1.89 bits per originalparameter — an average that is low mainly because half the experts were pruned — and the kept
experts are stored like IQ3_S (IQ2_S-IQ4_XS gate/up, IQ4_NL/Q2_0 down). So it is a ~3.5-bit
storage format on a pruned expert set, not a 1-bit one.
README.md:74says the same.What this changes:
point at.
vocab.jsonhas no merged CJKtokens and CJK goes through byte fallback (consistent with [tokenizer] O(n^2) BPE merge loop: an unbroken CJK run is ~10 s per 8K chars (1266x slower than needed) #268), and the measured result —
Chinese 0/10 correct vs English 10/10, unchanged across every ablation.
per token), selected with RCO on code/agentic/vision calibration data, and is documented as
weaker outside coding. Whether expert pruning specifically costs CJK is a different question
I have not tested.
I also cannot run the clean version of that test myself: the
CoderHF repo publishes only IQ1_M,and Q2_0 / IQ2_XS / IQ3_XXS / IQ3_S live in the separate unpruned original repo — different
weights, so not a quant-only ablation. Has anyone run the unpruned original and confirmed its
Chinese output? That would separate "pruning costs CJK" from "something in the CJK path is
broken across the board".
Apologies for the noise. The original measurements (the n=10 table, the nine ablations) are
unaffected.
lybhb8 commentedon Oct 2, 2026
Root cause found: expert pruning, not quantization — CJK works on the unpruned original
Follow-up to my correction above. I tested the unpruned original and the CJK failure does
not reproduce. This should redirect the investigation.
The experiment
Downloaded the unpruned
ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUFQ2_0 (37,623,740,192content-length), packed it withtools/iq_pack.py(needs--experts-bin; the engine rejects--mmap-expertswithout it andtells you so), ran the same prompt on a separate instance. Same engine 0.1.30, same box,
--kv int8,--spec 4, greedy.The original answers, all 10 runs, no degeneration:
通义千问and通义实验室both intact. Compare the Coder's我是通千and我是通（通通通通….Why this points at pruning
a CJK character,
通/千/义absent from the vocab keys. Both go through byte fallback.So my earlier byte-fallback observation holds for both and cannot explain the difference.
variable I ablated away in the original report is identical here.
selected with RCO on code / agentic / vision calibration data
(
setup.py:80,docs/DETAILS.md:167). Chinese is not in that set.stored at higher effective precision than the original's Q2_0 (IQ2_S–IQ4_XS / IQ4_NL–Q2_0
mix vs a flat 2-bit). If precision loss were the cause, the Coder should be better. It is
worse.
A second effect also shows up, possibly useful to you: at the same VRAM budget the Coder gets
8,520 expert slots vs the original's 13,193, with a decode hit rate of 82.9% vs 98.4%.
Pruning halves the expert pool but not the routing entropy, so a substantial fraction of the
256 kept experts are still rarely routed to.
What this does NOT establish
The comparison moves two variables at once:
So I can say the unpruned build is fine and the pruned one is not; I cannot strictly separate
"pruning" from "pruning + this storage format". Isolating it needs IQ3_S (3.5-bit,
unpruned, needs 64 GB RAM) or a higher-precision original. The observation in point 4 above
makes precision the less likely of the two, but it is not proof.
Suggested next step for you
If the Coder release is intended to serve more than code, the pruning calibration set may want
a language mix. A cheap check on your side would be re-running the RCO selection with some
multilingual calibration data and seeing whether Chinese survives; a CJK prompt in
bench/results/would also have caught this, sincegrep -rlP '[\x{4e00}-\x{9fff}]' bench/ tests/currently returns nothing at all.Happy to test more builds if you can point me at which ones matter — IQ3_S in particular would
close the remaining gap.
Niko1221 commentedon Oct 2, 2026
Thank you for the careful test, and your follow-up nails it: the unpruned Q2_0 gets 10/10 on the same engine and settings. This is a limitation of the Coder model, not the engine. The Coder keeps 256 of the 512 experts, chosen on code, agentic and vision data, so other languages and general writing are weaker; the experts Chinese text relies on are among the ones removed. The draft layer can't change greedy output (it only proposes, the model checks every token), and the tokenizer is the same as the original's. The next release says it plainly in the README and docs/MODELS.md: for Chinese and other non-code text, use the unpruned sizes (Q2_0, IQ2_XS, IQ3_S). Your CJK prompt set would make a good addition to
bench/if you'd like to open a PR. Closing.docs: the Coder is weaker outside code, Chinese/CJK text included; th…
docs: the Coder is weaker outside code, Chinese/CJK text included; th…
docs: the Coder is weaker outside code, Chinese/CJK text included; th…
docs: the Coder is weaker outside code, Chinese/CJK text included; th…