Repository navigation
Native CPU expert results depend on verifier width (plain/MTP divergence) #152
Description
Affected: Engine 0.1.24 (3ce2523), native i-quant expert packs. This is in the CPU expert path, so it is not specific to CUDA or HIP.
In native_expert.cpp, singleton expert groups use ggml vec_dot, while groups with nt >= 2 use custom AVX gate/up or IQ4_NL down kernels. These reductions produce slightly different FP32 results. The group size changes with verifier width, so the same weights and token activations can produce different expert outputs in plain and MTP runs. PR #43 documents the speed reason for the switch and its small AVX-versus-ggml error, but does not test equality across verifier widths.
Reproduction: Using one existing GSQ-RCO IQ3_S GGUF shard, take layer 0, expert 7 (IQ3_XXS gate/up, IQ4_NL down) and four fixed activations. Run the same native expert rows as four singleton calls, two calls of width 2, and one call of width 4; compare the FP32 outputs bitwise. Replaying the current mixed dispatch produced 9,566 differing cells against the singleton reference. With ggml vec_dot selected at every width, gate/up and down outputs matched bitwise for widths 1, 2 and 4. On a fixed 21,999-token prompt, the patched plain and MTP paths also generated the same 150 greedy token IDs; MTP accepted 93/130 drafts.
Expected: Changing the verifier or speculative width should not change target-model expert arithmetic for identical inputs.
Suggested fix: Select one arithmetic path independent of nt, and add a width 1/2/4 regression test. Using ggml vec_dot at every width is a conservative fix; retaining the multi-token AVX speed would require its result to be made width-invariant and validated separately.
Activity
Niko1221 commentedon Sep 29, 2026
Thanks @clapbr, confirmed:
native_expert.cpppicks the AVX multi-token kernels only for groups of 2 or more, so the verify width changes the rounding. For nowSTRATA_NO_IQ512=1 STRATA_NO_IQ256=1 STRATA_NO_IQ4NL=1gives width-invariant results (at some CPU speed). We'll choose one path for all widths, add your 1/2/4 test, and gate it.#152 as an opt-in (the owner's choice): STRATA_IQ_MT_MIN=1 makes CPU …
Engine 0.1.30: short prompts up to 28% faster (every expert streams f…
Niko1221 commentedon Sep 30, 2026
0.1.30 adds
STRATA_IQ_MT_MIN=1(in the config'senv): the CPU experts use the multi-token kernels for every group, so a token's expert rows round the same whatever the verify window held, and greedy output no longer depends on the drafting (checked bit for bit bynative_expert_parity). It costs 1-3% decode on IQ3_S with AVX-512 (IQ3_XXS +3%, the others within noise), so the default stays 0.1.29's rule. Thanks for the precise report!CPU experts: the multi-token kernels from one token on (Niko1221#152)…
CS-T speed: the file tier fetched in parallel, PCIe planning without …
Merge upstream main (30ec18e, engine 0.1.30): the server's GPU sharin…
Pipelined verify windows across a layer split (--pipeline-windows)
CPU expert kernels: gathered i-quant decodes, AVX-VNNI, IQ3_S for one…
native experts: IQ3_S takes the AVX-2 kernel for one token too where …
native experts: IQ3_S takes the AVX-2 kernel for one token too where …
Engine 0.1.30: short prompts up to 28% faster (every expert streams f…
CS-T speed: the file tier fetched in parallel, PCIe planning without …
native experts: IQ3_S takes the AVX-2 kernel for one token too where …
29 remaining items
serve の tail skip: 飛ばすかをエントリーごとに決める
入れ替え: 文章の決まった位置で、確定した行のルーティングから行う
GPU のエキスパート: CPU と同じ決まった順と算術で計算する（Niko1221#152）
CPU のエキスパート: GPU と同じビットになる行で計算する（Niko1221#152）
エキスパート: IQ4_XS と Q2_0 の gate/up も CPU と GPU で同じビットにする（Niko1221#152）
上流の XeStrata main（180d0776、xe0.1.45-1）を取り込む
890M の既定: エキスパートの行は fork のロードを先に出す形、serve の入れ替えは窓ごと
fix(helper): CPU reference worker computes an expert for all its toke…
docs(CONCURRENCY-RADIX): review-2 - concurrency is SHIPPED production…
docs(CONCURRENCY-RADIX): review-3 - dedup G3 row, fix 4 wrong lines, …
CPU trellis kernels: AVX2 / AVX-512 decode-and-dot for TQ2_T / TQK6 /…
spec: STRATA_MTP_ADAPTIVE=1, the MTP window's own size from the draft…
native experts: window-invariant CPU arithmetic for STRATA_SPEC_PROB …
native experts: window-invariant CPU arithmetic for STRATA_SPEC_PROB …
エキスパートの層: PlainModel の入れ替えの規則を Qwen3.8-Flash-Next に揃え、部品を共有に