Skip to content

CPU i-quant kernels from upstream: IQ4_XS on AVX-2, the gathered IQ3 decode, AVX-VNNI, an AVX2 Q8_K quantizer - #17

Closed
txiki739 wants to merge 13 commits into
eddoursul:customfrom
txiki739:cpu-iquant-kernels
Closed

CPU i-quant kernels from upstream: IQ4_XS on AVX-2, the gathered IQ3 decode, AVX-VNNI, an AVX2 Q8_K quantizer#17
txiki739 wants to merge 13 commits into
eddoursul:customfrom
txiki739:cpu-iquant-kernels

Conversation

@txiki739

@txiki739 txiki739 commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Upstream's (Niko1221/Strata) CPU expert kernels, picked onto this tree. None of this is our code: each commit
keeps its author, and the original commits are named in their messages (cherry picked from ...). We only added three
small glue commits so they build here. Sending them so you know they exist and what they gave on this engine.

commit(s) author what
IQ4_XS expert rows on the AVX-2 path (upstream Niko1221#415) pipeob0 IQ4_XS had no multi-token AVX-2 kernel: its one layer went token by token
IQ4_XS's AVX-2 kernel on AVX-2 CPUs only (Niko1221#415 follow-up) Niko1221 AVX-512 CPUs keep ggml-cpu's rounding
the IQ2 sign table without a variable shift (Niko1221#391) / at compile time Demetree Kallergis, Niko1221 no BMI2 shlx and no static constructor in an AVX2 file (crashes on AVX-only CPUs)
gather the AVX2 codebook from the grid table (STRATA_IQ256_GATHER) Yannick Loth, Niko1221 the IQ3 grid words with one gather instead of eight scalar loads
the gathered decode on the cores where it is faster (PR Niko1221#863) Francesco Albano (Hardin22) its own format variant, chosen once per call
an AVX2 Q8_K activation quantizer (PR Niko1221#851) Francesco Albano (Hardin22) byte-identical to ggml's
AVX-VNNI forms of the AVX-2 i-quant, IQ4_NL and Q2_0 rows Francesco Albano (Hardin22) Alder Lake and later; bit-identical integer sums
IQ3_S takes the AVX-2 kernel for one token too where its grid is gathered Francesco Albano (Hardin22) Intel cores
cpu_isa_cap() from upstream (e82947f), <cstring>, native_expert_parity without SIGILL on Zen 3 us glue

Measured on this engine (Ryzen 7 5700X, Zen 3, UD-IQ4_XS, six-prompt decode mean)

Where this sits in our setup

Part of what we run on a Ryzen 7 5700X + 128 GB DDR4 + RTX 3090 (+ RTX 5060 Ti 16 GB), Linux, on top of your custom
(with #2, #5 and #6). Six-prompt decode mean, UD-IQ4_XS: 77.3 tok/s on custom with those fixes, 92.9 on the
3090 alone with everything in this series, 127.4 with both cards; UD-Q4_K_XL 71.2 / 90.6; an 18-step agent
session 86.2 / 110.9 tok/s; a 200K prompt read in 82.9 s and decoded at 97.6 tok/s right after it (both cards).

This series

Each applies on custom at 3a19944 and builds (the engine target; every target with the tools fix). Stacked ones say so at the top. Ours unless noted; the ported ones name their authors.

Tested

Smoke runs on custom at 3a19944 (RTX 3090 or both cards, UD-IQ4_XS, greedy, a Spanish chat answer and a code prompt): with STRATA_IQ256_GATHER=1 76.6 / 82.5 -> 78.7 / 85.7 tok/s (+3 / +4%), the same tokens.

On custom today, the same smoke runs: RTX 3090 76.6 / 82.5 tok/s -> 92.0 / 98.4 with our full engine and current config; both cards 72.4 / 119.9 -> 110.5 / 132.7.

demetree and others added 13 commits October 12, 2026 08:18
@demetree @txiki739
iq_avx2: build the IQ2 sign table without a variable shift
dee6f49 
A namespace-scope static initializer shifted a byte into place with << (8 * k), which is a
variable shift.  MSVC compiling this file with /arch:AVX2 emits BMI2 shlx for that - it assumes
anything with AVX2 also has BMI2, true from Haswell on and false of everything before it.

Because the table is static, the constructor runs before main(), so this faulted at PROCESS START:
strata.exe exited 0xC000001D with no output at all, not even for --help.  One shlx in the whole
linked binary was enough to make the engine unstartable on any pre-Haswell CPU under MSVC.

Storing the byte directly is the same 64-bit value on the little-endian targets this file is for,
and leaves no variable shift for the compiler to widen.

Found by building the engine for sm_61 on a Sandy Bridge-E (Xeon E5-2680) with
STRATA_EXPERIMENTAL_SM60=ON, which 0.1.31 added (Niko1221#236).  Verified by disassembly: no shlx/shrx/sarx/
mulx/andn/bzhi/pext/pdep anywhere in strata.exe afterwards, and --help now runs.

This is independent of any AVX1 work - it is a plain startup crash on the CPU this was built on.

(cherry picked from commit 3fcaf03)
(cherry picked from commit dd0ce1e)
@pipeob0 @txiki739
IQ4_XS expert rows on the AVX-2 path: the multi-token kernel this for…
8570619 
…mat never had

IQ4_XS (23) was missing from iq256_supported, so on an AVX-2-only CPU its gate/up expert rows
fell back to ggml's per-token vec_dot while every other i-quant format got the multi-token
kernel that decodes the weights once for the whole verify window.

Fmt32<23> reuses the 16-value codebook IQ4_NL already has. Each of the eight 32-value sub-blocks
carries its own 6-bit scale (two nibbles of scales_l plus two bits of scales_h), used signed as
(ls - 32): the scale folds into the int16 operand of madd_epi16 instead of becoming a float
multiply per sub-block, and the codebook sign is carried the way iq4nl_rows does it (|w| as the
unsigned maddubs operand, w's sign applied to the activation).

The gate in native_expert.cpp and in native_expert_parity cannot be iq512_supported alone:
IQ4_XS has an AVX-2 kernel and no AVX-512 one, so on an AVX-512 CPU the widened gate must still
pick each kernel only for the formats it implements - falling through an empty switch would
leave ff unwritten rather than falling back to ggml-cpu. Widening the parity gate is what makes
the format tested at all: with iq512_supported alone the tool skipped it on every CPU.

(cherry picked from commit e7b415c)
@Niko1221 @claude @txiki739
IQ4_XS's AVX-2 kernel on AVX-2 CPUs only; native_expert_parity skips …
6d10613 
…the AVX-512 check it has no kernel for (Niko1221#415 follow-up)

- The engine's `avx2` switch is on for AVX-512 CPUs too, so the widened gate sent an AVX-512 CPU's IQ4_XS gate/up rows
  to the new AVX-2 kernel instead of ggml-cpu: other rounding on CPUs that Niko1221#415 did not mean to change. A format with
  only an AVX-2 kernel now takes it on CPUs without AVX-512; everything else is dispatched as before.
- native_expert_parity ran its AVX-512 check for IQ4_XS on an AVX-512 CPU: iq512_rows has no case for it, so the
  rows stayed zero and the check failed. Each check runs only for the formats its kernel implements; the AVX-2 one
  runs on AVX-512 CPUs too, so this PC tests the new kernel.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
(cherry picked from commit 5696360)
@Niko1221 @claude @txiki739
iq_avx2: the IQ2 sign table at compile time (no static constructor in…
95532c3 
… an AVX2 file)

Niko1221#391 removed MSVC's BMI2 shlx from the table's constructor; GCC still vectorised the same loop into
AVX2 (vpbroadcastb) and ran it before main() on an AVX-only Xeon E5 (found by rwkeyes' Strata_Dirigo
fork, 6dd37a2, which made it constexpr). constexpr here as well, deriving ksigns_iq2xs[i] (the 7 bits
plus an even-parity bit) so the constant expression does not read a non-constexpr array. Same 128
values; the file now has no runtime initializer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 016ea2e)
@yannickloth @txiki739
iq: gather the AVX2 codebook from the grid table (STRATA_IQ256_GATHER)
11a584f 
The two IQ3 gate/up formats decode each 32-value half with eight scalar loads
assembled by _mm256_set_epi32.  A scratch build that substitutes a constant for
the grid vector shows that assembly is ~half the single-core kernel (IQ3_XXS
248 -> 130 us, IQ3_S 357 -> 140 us), and the two formats are ~58% of the
gate/up lookup work on the Swift 1.5 IQ3_XXS pack.

Add an opt-in AVX2 gather for IQ3_XXS (iq3xxs_grid) and IQ3_S (iq3s_grid),
the mirror of the AVX-512 kernels' STRATA_IQ_GATHER.  On the i7-12850HX
(Alder Lake, no AVX-512) it is bit-identical - a direct scalar-vs-gather output
dump matches for all five gate/up formats, native_expert_parity stays green -
and 1.32-1.34x on the two formats isolated, +1.0% serve end to end (on >= off in
5 of 5 interleaved pairs).  The pool is flat in the deterministic --stats (the
15-worker pool is not decode-bound), so it is a per-core win that compounds.

Default off, like the AVX-512 one: AVX2 gather throughput is microarchitecture-
dependent.  IQ2_XXS / IQ2_XS / IQ2_S use 64-bit grids, where an AVX2 i64 gather
measured a wash, and are unchanged.  The AVX-512 path is untouched.

(cherry picked from commit dd24a55)
@Niko1221 @claude @txiki739
iq_avx2: STRATA_IQ256_GATHER=0 keeps the scalar grid assembly (parse …
01b0c4b 
…the value)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 3a64d02)
@txiki739 @claude
native_expert_parity: the AVX-512 rows only on a CPU that runs them (…
b20d6b5 
…SIGILL on Zen 3 before the first line)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Hardin22 @txiki739
cpu experts: an AVX2 Q8_K activation quantizer for the native i-quant…
ce1360e 
… layers

Every i-quant expert row the CPU computes takes its activations as Q8_K: the
token's hidden state for gate/up (native_quant_act, once per token and layer,
on the host before the pool starts), and each expert's FFN output for down
(native_quant_h, per expert and token, in the pool). On x86, ggml-cpu's
quantize_row_q8_K is the scalar reference (arch/x86/quants.c calls
quantize_row_q8_K_ref).

q8k_quant_avx2 does the same arithmetic in AVX2: the same max with the first
value of the largest magnitude keeping its sign, the same iscale, one multiply
and the same 1.5*2^23 rounding add per value (kept apart on GCC, which would
fuse them into an FMA), the same clamp and int8 store, the same bsums, and
zero blocks left as the reference leaves them. The bytes are identical.

- Gated on cpu_avx2_ok() (iq_avx2.cpp is compiled for AVX2), so an AVX-only
  CPU or STRATA_FORCE_ISA=avx keeps ggml's; STRATA_NO_Q8K_AVX2=1 keeps ggml's
  on any CPU.
- q8k_quant_parity (CTest, CPU only): the AVX2 copy, ggml-cpu's from_float and
  native_quant_act / native_quant_h against quantize_row_q8_K_ref, memcmp of
  the whole rows at 256, 512, 2560 and 6144 values, over random, scaled,
  sparse, zero, tied-maximum, extreme, denormal, grid-tie, NaN and inf rows;
  also with STRATA_FORCE_ISA=avx. --bench times both.

i9-14900KF (P-core), a 2560-value row: 2141 ns -> 891 ns (2.4x).

End to end, Swift 1.5 IQ3_XXS at 160K, greedy, 4 prompts x 3 rounds, decode tok/s:
- RTX 4060 Ti + RTX 5080 layer split with the resident RAM mode (Niko1221#848), two
  interleaved pairs: 66.0 / 67.6 -> 68.5 / 69.0 (+2.9%); the host's actq per
  window 0.35 -> 0.15 ms of a 36.5 ms window; code and Italian prose faster in
  both pairs, English prose in one (equal in the other).
- RTX 5080 alone: actq 0.50 -> 0.20 ms of a 70-150 ms window, within the noise.

(cherry picked from commit f507fcb)
@txiki739 @claude
expert_layout: cpu_isa_cap() and cpu_avx2_ok() from upstream (e82947f)…
fc934e3 
…, which PR Niko1221#851's quantizer gates on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Hardin22 @txiki739
iq_avx2: the gathered IQ3_XXS / IQ3_S / IQ2_S decode on the cores whe…
20785b1 
…re it is faster (STRATA_IQ256_GATHER auto)

dd24a55 reads the IQ3 grid words with one gather under STRATA_IQ256_GATHER=1, on every core, and keeps it off by
default because AVX2 gather throughput depends on the core. This makes the gathered decode its own Fmt32 (118, 121,
122), chosen once per call instead of a branch in every decode, and takes it further:

- IQ3_XXS: the four even-sign masks by one 64-bit gather as well;
- IQ3_S: the 9th index bit by one sllv and one and; the sign bytes by one broadcast and one shuffle;
- IQ2_S: the four 10-bit grid indices built in a register and read by one 64-bit gather, the signs as IQ3_S's.

Each is its format's scalar decode bit for bit: the same words land in the same lanes. IQ2_XXS measured 4% slower
gathered and IQ2_XS has its own sign decode, so both keep the scalar one.

STRATA_IQ256_GATHER: unset = auto, 0 = the scalar decode everywhere (as before), 1 = the gathered one everywhere.
Auto gathers on a thread whose core is an Intel performance core of Alder Lake / Sapphire Rapids or newer:

- GenuineIntel with AVX-VNNI (CPUID 7.1:EAX[4]) as the generation marker. That leaves out Haswell/Broadwell (slow
  gathers) and Skylake through Ice Lake / Tiger Lake (gathers slowed by the Downfall/GDS microcode);
- not the E-core-only parts, which are not hybrid: Alder Lake-N / Twin Lake, Grand Ridge, Sierra Forest and
  Clearwater Forest, by model;
- on a hybrid CPU only a thread on a P-core (CPUID 1Ah core type 40h): the E-cores gather slower than they assemble.

AMD keeps the scalar decode (Zen 2/3 gather slowly; Zen 4/5 take the AVX-512 kernels). STRATA_FORCE_ISA turns auto
off. The probes are in expert_layout.cpp, which is not compiled for AVX2, and the core type is asked once per thread
(the pool pins its workers).

Measured on an i9-14900KF, gate/up rows of one expert at three tokens on one P-core, scalar -> gathered: IQ3_S 0.40
-> 0.27 ms, IQ2_S 0.31 -> 0.27, IQ3_XXS 0.26 -> 0.24. On its E-cores the gathered decode was slower.

iq_avx2.cpp has no runtime initializer again (016ea2e): the switch and STRATA_IQ_PREFETCH are read on first use.

iq_avx2_parity, a CPU-only CTest: random blocks (every grid index and sign pattern occurs) for IQ2_XXS, IQ2_XS, IQ2_S,
IQ3_XXS, IQ3_S and IQ4_XS gate/up rows and IQ4_NL / Q2_0 down rows; every variant, 1..8 tokens, each token's rows bit
for bit the scalar variant's rows for that token alone; the scalar rows against ggml-cpu's vec_dot (Q2_0: a double
reference). --bench: ms per expert on one thread, each variant, ggml's per-token dot and the engine's dispatch, over
more expert blobs than an L3 holds.

(cherry picked from commit bc48c52)
@Hardin22 @txiki739
cpu kernels: AVX-VNNI forms of the AVX-2 i-quant, IQ4_NL and Q2_0 row…
61389d5 
…s (cpu_avxvnni_ok)

Alder Lake, Sapphire Rapids and later have AVX-VNNI on every core: vpdpwssd does the madd and the add of an i-quant
row in one uop, and vpdpbusd the maddubs + madd(ones) of the IQ4_NL and Q2_0 down rows. The sums are integer, so the
results are bit for bit those of the AVX2 forms: vpdpwssd wraps like madd + add, and the maddubs it replaces in the
down rows cannot saturate (IQ4_NL |w| <= 127, Q2_0 codes 0..3).

The row kernels move to iq_avx2_rows.inl and q2_avx2_rows.inl, which iq_avx2.cpp and q2_avx2.cpp include twice: once
as they were (namespace plain) and once with AVX-VNNI (namespace vnni). In the second copy every function is declared
with target("avxvnni") under GCC/Clang, so the intrinsics compile there and nowhere else: neither file gets
-mavxvnni, which would let the compiler use the extension in the plain path. MSVC takes the intrinsics under
/arch:AVX2. CMake checks that the compiler has them (GCC 11, Clang 12) and otherwise builds without the copies
(STRATA_AVXVNNI=0).

cpu_avxvnni_ok() in expert_layout.cpp, which is not compiled for AVX2: cpu_avx2_ok() and CPUID 7.1:EAX[4], off under
STRATA_FORCE_ISA (a CPU that stops at AVX2), off in a STRATA_ISA_FLOOR build, and STRATA_NO_AVXVNNI=1 turns it off.
The VNNI copies are reached only through the AVX2 kernels' entry points, which the engine already calls only behind
cpu_avx2_ok().

Measured on an i9-14900KF on top of the gathered decode: another 4-10% on a P-core's i-quant gate/up rows, 15-25% on
an E-core's Q2_0 down rows.

iq_avx2_parity runs the VNNI variants as well (iq4nl256_down_rows_v, q2_0_gguf_rows_multi_avx2_v): every combination
bit for bit against the scalar one-token rows, 1..8 tokens.

(cherry picked from commit e9802b4)
@Hardin22 @txiki739
native experts: IQ3_S takes the AVX-2 kernel for one token too where …
0d2c59b 
…its grid is gathered

token. With the gathered decode that no longer holds for IQ3_S on the Intel cores that gather it: an expert's gate/up
rows for one token on an i9-14900KF P-core take 0.34 ms with ggml's dot and 0.19 ms with the AVX-2 kernel. And in
decode most CPU experts serve one token of the window.

So on a CPU without AVX-512 where cpu_gather_fast() holds (Intel, Alder Lake or later, see the gather commit) and
STRATA_IQ256_GATHER is not 0, IQ3_S takes the multi-token kernel from one token on. This only pays together with the
gathered decode: on the same CPU's E-cores, which keep the scalar decode, it is 0.71 -> 0.68 ms, about even. That is
why it follows the gather's machine-wide rule and not the per-thread one. Measured in the engine with Swift 1.5
IQ3_XXS (whose IQ3_S layers this reaches): the CPU expert time per verify window -4%.

This changes results. A lone token's IQ3_S gate/up rows now round like the multi-token kernel's (the order of the
float additions), no longer like ggml's dot. They are the same rows the token gets in a group, so on these CPUs IQ3_S
experts no longer depend on how many drafts a window held (Niko1221#152), and every core rounds them alike. Down rows and
the other formats keep the default rule. STRATA_IQ_MT_MIN set (2 is the old default) keeps its rule for every format.

The decision is native_gu_mt_min(): every probe is in expert_layout.cpp and cpu_avx2_ok() comes first. Nothing
compiled for AVX2 runs before that check, including on an older-CPU build (STRATA_ISA_FLOOR) on an AVX-only CPU.
iq_avx2_parity checks native_gu_rows wherever native_gu_mt_min is 1: each token's rows must match alone and in groups
of 2..8. docs/DETAILS.md notes the new default next to STRATA_IQ_MT_MIN.

(cherry picked from commit 695a75b)
@txiki739 @claude
expert_layout: <cstring> for cpu_name's memcpy (PR Niko1221#863 on th…
2ea98f0 
…is tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 12, 2026
Merged
Merged
Merged
Closed
Merged
Closed
Merged
Merged
Merged
@txiki739

txiki739 commented Oct 12, 2026

Copy link
Copy Markdown
Author

Closing: these kernels came in with your upstream 0.1.43 merge (e3727cb): Niko1221#391, Niko1221#415, Niko1221#851, Niko1221#863, Niko1221#930, the gathered IQ3 decode and the AVX-VNNI forms are all in custom now. The numbers above still describe what they give on Zen 3.

@txiki739 txiki739 closed this Oct 12, 2026
@eddoursul

eddoursul commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Thank you for collecting these, and right: the kernels came with the 0.1.43 merge. What custom still lacked was the CMake side: without the AVX-VNNI probe the copies were never compiled, and the two parity tests were not built. Both are in now (ad6e0ec), with your native_expert_parity guard, and your branch's head is recorded in custom with -s ours (b9e5fab). On this Zen 5 the AVX-VNNI IQ4_NL down rows of 2-4 tokens take 5-9% less time on one thread; end to end within 0.2% (IQ3_S's only AVX-2 rows here; the rest run the AVX-512 kernels), window hashes unchanged.

(Measured on a Ryzen 7 9700X, 160 GB DDR5-5600, RTX 3090 + RTX 5070 Ti as the second expert tier, Windows 11; IQ3_S and UD-Q4_K_XL with this branch's configs, fixed text through --spec-follow, 2 rounds in both orders.)

eddoursul added a commit that referenced this pull request Oct 12, 2026
@eddoursul @claude
CPU experts: upstream's AVX-VNNI probe in CMake (the AVX-2 rows' vpdp…
ad6e0ec 
…wssd / vpdpbusd copies, IQ4_NL down rows of 2+ tokens on this CPU), iq_avx2_parity and q8k_quant_parity built, native_expert_parity's AVX-512 rows only where the CPU runs them

From PR #17 (txiki739; upstream's b211ba1 by Francesco Albano): its kernels and the rest are in this branch with
upstream 0.1.43.  iq_avx2_parity on the Ryzen 7 9700X: every variant bitwise the scalar rows; IQ4_NL down rows of 2-4
tokens 5-9% less time on one thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
eddoursul added a commit that referenced this pull request Oct 12, 2026
@eddoursul @claude
Merge pull request #17 from txiki739/cpu-iquant-kernels, its content …
b9e5fab 
…in this branch with upstream 0.1.43 and ad6e0ec

Upstream's CPU expert kernel files (IQ4_XS on AVX-2, the gathered IQ3 decode, AVX-VNNI, the AVX2 Q8_K quantizer, the
IQ3_S one-token kernel) came with the 0.1.43 merge in their newer form; the pull request's CMake side (the AVX-VNNI
probe, iq_avx2_parity, q8k_quant_parity) and its native_expert_parity guard are in ad6e0ec.  Merged with -s ours: the
tree is unchanged; the commit records the pull request as merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@txiki739
txiki739 deleted the cpu-iquant-kernels branch October 13, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@txiki739 @eddoursul @demetree @pipeob0 @Niko1221 @yannickloth @Hardin22