[hotschmoe]

Problem and result

The SYCL lane kernels that read the routed experts (native_gu_kernel / native_down_kernel, 8 lanes a row) are ALU-bound on an Arc Pro B70, not memory-bound: 280-480 instructions per 32-weight part for its 8 dp4a, reading the experts at 19-44% of DRAM bandwidth. Most of it is the dpct per-byte emulation of byte_level_permute in the Q2_0 dot, the compare/select chain of the IQ4_NL codebook lookup, the popcount/multiply ksigns unpacking, and 64-bit kbx * sizeof(block) products every iteration.

STRATA_IQ_DECODE_FAST=1 switches the seven IQ expert formats (IQ2_XXS, IQ2_XS, IQ3_XXS, IQ3_S, IQ2_S gate/up; IQ4_NL, Q2_0 down) to kernels that produce the same bits with fewer instructions. Off by default; without the flag the engine runs exactly the code it ran before.

Result on 0.2.0-beta, two B70s, one stream: decode 70.5 / 73.7 -> 76.7 / 80.1 tok/s (+8.7%), with output tokens and draft counts identical, so the gain is kernel time alone (verify window 39.5 / 40.2 -> 36.3 / 36.9 ms). Prompt reading unchanged.

Related discussion

Feature issue #2801 (closed in the 0.2.0 tracker reset). No maintainer agreement on scope yet; opt-in and SYCL-only so the default path stays byte-identical. Happy to change the switch name or drop formats.

Approach

• sycl/src/kernels/cuda/iq_kernels.dp.cpp: IqFast<TY> decode traits whose prep yields the same integers as Multi<TY>::prep / the vec_dot_*_q8_1 dot and reuses the same acts and finish; ksigns masks from a 2 KB table, the IQ4_NL codebook through four 256-entry position tables, Q2_0 fields spread with shifts and masks, aligned 8-byte loads. rows_dot_fast keeps the k order, per-entry float sum order and lanes_sum of the current dots, so the bits cannot move. Two rows per lane in 16-lane sub-groups. One hook line at the top of launch_gu_port / launch_down_port that returns at once when the latched flag is off. STRATA_IQ_DECODE_FAST_FMTS=<ggml ids> restricts it for an A/B.
• sycl/src/kernels/cuda/iq_decode_tables.inc (new): the two tables with the formulas that generate them.
• sycl/src/kernels/native_grouped_parity.cpp: --iq mode, fast against current, bitwise, every IQ gate/up format with both IQ down formats, groups of 1..6 entries, scattered destinations, stale scratch, rows a call must not write. sycl/CMakeLists.txt: ctest native_grouped_parity_iq_decode.
• docs/INTEL.md: a section on the flag.

Left alone: include/, src/, CUDA and HIP, every kernel the default path runs, the K-quant / Q5_1 / Q8_0 expert formats, the prompt path.

Validation

Setup: 0.2.0-beta main e88ee39 against this branch rebased on it. 2x Intel Arc Pro B70 32 GB, kernel 7.1, Compute Runtime 26.22.38646.4 at run time; built with the repository dev image (sycl/tools/Dockerfile on the current llama-sycl-b70:stable, IGC 2.41.5; an older cached base with IGC 2.36.3 cannot build 0.2.0, see #3219), AOT bmg-g31. Model Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS, --layer-split 30 --prefill 8192 --max-context 65536 --kv int8 --spec 4 --mtp <BF16 draft layer> --mtp-max-t 4 --spec-min-p 0 --stream-experts --expert-cache auto, one slot, greedy, two fixed coding prompts with 512-token replies, a fresh engine per arm, then fresh 4K and 32K prompts. Coherence check and per-card health after every stop.

Passed:

• native_grouped_parity --iq (0.2.0-beta + this branch, head 7035777, one B70): 0 failures, every case bitwise equal. Plain native_grouped_parity: 0 failures.
• Engine A/B (tok/s, verify window ms, per prompt):

	prompt 1	prompt 2	4K / 32K prompt tok/s
main e88ee39	70.6 (39.4 ms)	73.9 (40.0 ms)	1,219 / 2,078
this branch, flag unset	70.5 (39.5 ms)	73.7 (40.2 ms)	1,219 / 2,082
STRATA_IQ_DECODE_FAST=1	76.7 (36.3 ms)	80.1 (36.9 ms)	1,224 / 2,084

• Flag unset and set: output SHAs and draft counts (549/328, 525/339) identical to main on both prompts. Coherence true and per-card health passed after every stop.

Failed: none observed.

Not run: two concurrent streams (main refuses --batch-mtp on a layer split), --max-context 200000 (a plain two-card layer split does not fit all 24,576 experts), CUDA/HIP/Metal builds (no file touched), Arc A-series cards, ctest as a whole (the parity binaries were run directly).

Limits

• Only the 8-lane kernels get the fast form; other STRATA_GU_LANES / STRATA_DOWN_LANES keep the current kernels.
• The docs/INTEL.md section quotes the earlier measurement on 0.1.x main 296b59d (79.5 -> 85.6 tok/s); I can swap in the 0.2.0 numbers above if you prefer.
• Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.

🤖 Generated with Claude Code

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[hotschmoe]

/reopen

Rebased on 0.2.0-beta (main e88ee39) and re-validated on two Arc Pro B70s; the description has the template and the new numbers. The rebased branch goes up right after the reopen (GitHub cannot reopen a pull request whose branch was force-pushed while it was closed).

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.

[deriknel]

Tested on 2x Arc Pro B70 (CachyOS, kernel 7.2.11, xe) with Swift 1.5 Flash-Next IQ2_XS: layer split 26, every expert in VRAM, int8 KV, MTP spec 4, 262144 context. The image is v0.2.0-beta (3d1fb48) plus our SYCL patches, including #2804 (STRATA_MMVQ_KREG=1) and #2808. Each arm ran on a fresh engine, two rounds. Greedy, 5 fixed prompts, decode tok/s:

	prose	code	maths	edit	story	mean
flag unset	107.9	147.3	134.7	143.7	81.4	123.0
STRATA_IQ_DECODE_FAST=1	111.4	151.7	138.9	148.1	83.7	126.8

That is +3.1%, with every prompt between +2.8% and +3.2%. The second round matched within 0.2 tok/s. The text was identical on all 5 prompts in both rounds, and the MTP acceptance rates did not move. Prompt reads at ~1K, ~4.5K, ~33K and ~148K tokens were unchanged, as expected for a decode-only change.

With #3248 and #2810 also on, the flag still adds its share: the three together give 136.7 tok/s mean (+11.2%). No xe resets or page faults over the whole run. The kernel suite passes (39 tests, the 1 failure is ple_parity, which needs a Q2_0 model file we don't have).

We carry this as a local patch with the flag on until it merges.

We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.