Skip to content

[Feature]: SYCL: cheaper IQ weight decodes in the grouped expert kernels (opt-in, same bits) #2801

Closed as not planned

Description

@hotschmoe

What do you want to do?

Decode faster on Intel cards whose expert kernels are instruction-bound rather than bandwidth-bound.

On an Arc Pro B70 (SYCL port, Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, two cards with a layer split) an ISA dump of the
lane kernels that read the routed experts (native_gu_kernel / native_down_kernel in
sycl/src/kernels/cuda/iq_kernels.dp.cpp) shows 280-480 instructions per 32-weight part for the part's 8 dp4a: the
Q2_0 down dot spends ~300 of its 476 on the dpct per-byte emulation of byte_level_permute, the IQ4_NL dot ~100 of
279 on the compare/select codebook lookup, and every format pays popcount/multiply ksigns unpacking and 64-bit
kbx * sizeof(block) address products each iteration. The kernels read the experts at 19-44% of the card's DRAM
bandwidth, so decode is slower than the memory system allows.

I would like an opt-in flag (STRATA_IQ_DECODE_FAST=1) that switches the seven IQ expert formats to kernels
producing exactly the same bits with fewer instructions: table-driven ksigns masks, the signed grid byte in one xor
and one add, Q2_0 fields spread by shifts and masks, the IQ4_NL codebook through small position tables, a loop that
walks block pointers, two rows per lane in 16-lane sub-groups, and a multi-entry form for IQ2_XXS.

Measured on two B70 (32 GB, kernel 7.1, Compute Runtime 26.22.38646.4, oneAPI 2026.1), main against a branch with
these kernels, --layer-split 30 --max-context 65536 --kv int8 --spec 4 with the BF16 draft layer, greedy, a fixed
512-token coding reply: decode 79.5 -> 85.6 tok/s median (+7.7%), 76.0 -> 81.4 at a 40,796-token context, output
tokens and draft acceptance counts identical, prompt speed unchanged. A grouped-call microbenchmark on one card put
the Q2_0 down product at 63.7 -> 26.4 us per call and the IQ2_XXS/Q2_0 layers at 114.8 -> 59.8.

Workaround today: none; the kernels are what they are. The default path should stay byte-identical, so the change
must be opt-in and come with a parity test (native_grouped_parity --iq) that compares the new kernels against the
current ones bit for bit on random blobs for every IQ gate/up and down format.

Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.

Activity

  1. Niko1221 commented on Oct 19, 2026

    @Niko1221
    Owner

    Thanks for taking the time to report this.

    We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

    To reopen it takes about two minutes:

    1. Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
    2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
    3. Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
    4. Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

    The rules are in CONTRIBUTING.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions