[xiangyuT]

Problem and result

IQ4_XS × Q8_1 decode repeatedly loads headers, packed weights and adjacent activation blocks. This patch shares those loads across quantization groups/token columns and uses an exact bit-select lookup.

2026-10-20 retest on a single Arc Pro B70, 32 GiB container RAM limit, upstream 0.2.0-beta:
upstream/common frontend e88ee39d1d6be6e3364183af3b39b62a7ed6f211; independent candidate 635ebb20c13255250a5b9fc79f2592a8256e5268.
All three lengths are complete.
These are standalone-PR results; no other preview optimizations are enabled.

Input / output	TTFT upstream → enabled (s)	Improvement	TPOT upstream → enabled (ms/token)	Improvement
4096 / 128	3.151388 → 3.150641	+0.024%	14.730184 → 13.866511	+5.864%
8192 / 256	6.377553 → 6.376644	+0.014%	13.858997 → 13.072603	+5.675%
32768 / 256	27.296890 → 27.289905	+0.026%	14.392408 → 13.585960	+5.603%

Positive improvement means lower latency: mean of three request-wise 100 × (1 − candidate / mean(upstream-before, upstream-after)) values.
The displayed upstream time averages both bracketing blocks. TTFT is request-to-first nonempty SSE text;
TPOT is first-to-last nonempty SSE text arrival divided by engine-generated tokens minus one.
Tiny changes near baseline variation do not establish a meaningful improvement.

Related discussion

Source status (2026-10-20): source branch feat/sycl-iq4-mmvq now points to the tested candidate 635ebb20c13255250a5b9fc79f2592a8256e5268. This PR remains closed; GitHub still reports PR head e5c0c9c69d38c9469c185ba67c0f7d61e0ecc564. All new measurements and validation in this description apply to the tested candidate. Reopening remains for maintainer review.

This updates the existing PR after the 0.2.0-beta reset and current contribution requirements; it is not a competing submission.
Scope and reopening remain for maintainer review in this thread. Historical pre-beta evidence is folded below.

Related MMVQ implementation: #2804 / #2803 select KREG kernels at the same entry points. Coordinate shared dispatch hooks with this pair-reuse/lookup path.

Approach

Battlemage only, K=2560/6144 and T=1–6, with alignment/row/shape guards; pair reuse and its Q8-split child require T≤4. The single-column K=2560/N=640 case retains its existing dispatch. Q8 splitting is further limited to selected shapes.

New paths are now opt-in. Every control below is disabled when unset or 0.
Set controls before starting the engine and restart after changes because values are cached/read during initialization.
Only the controls for this PR are set to 1 in the measured candidate; pure upstream and the candidate-default check leave them unset.

Control	With 1
STRATA_SYCL_IQ4_PAIR_REUSE	Pair reuse; also permits its Q8-split child.
STRATA_SYCL_IQ4_Q8_SPLIT	Adjacent Q8 blocks/scales; requires pair reuse.
STRATA_SYCL_IQ4_LOOKUP	Independent bit-select lookup when pair reuse is not selected.

To disable the complete change, leave all three unset or set both PAIR_REUSE and LOOKUP to 0. Q8_SPLIT has no effect without PAIR_REUSE.

Validation

Passed

• Exact-source SYCL Release/AOT bmg-g31 engine build. No non-SYCL backend files are changed.
• 78 IQ4 full-output hash rows match upstream, default-off and enabled execution; per-column byte comparisons and output guards pass.
• Fresh-process serving: all seven corresponding requests (one short warmup, three full-input warmups, three measured requests) have identical engine token IDs and text across upstream-before / enabled candidate / upstream-after at every length. The extra 4K candidate-default block also matches all seven requests.
• Timing blocks restart after an excluded INT8 calibration process, read the same persisted crossover file, and verify its hash is unchanged. Warmup/calibrating/default-functional blocks are excluded from performance.

Sanitized operator output:

iq4-baseline.log: RESULT cases_outputs=646482 bit_diffs=0 nonfinite=0 failures=0
iq4-default.log: RESULT cases_outputs=646482 bit_diffs=0 nonfinite=0 failures=0
iq4-enabled.log: RESULT cases_outputs=646482 bit_diffs=0 nonfinite=0 failures=0

Failed

The initial upstream/default 4K warmup comparison stopped on different output between a first-time calibrating upstream process and subsequent cached processes. All three measured requests matched, but the warmup mismatch was preserved. Restarting the same unmodified upstream binary with the persisted cache matched all seven candidate-default requests. The current comparison excludes that attempt and uses restarted, cached processes throughout. The upstream cache stores a rounded crossover; the controlled result establishes the initialization-state difference, without isolating every numerical cause.

Not run

Runtime scope for this retest is the single B70/Linux/AOT setup below. Other GPUs, Windows, multi-GPU serving, JIT and whole-engine native recording were not revalidated.
Old cross-target compilation, broad prompt/logit corpora and historical microbenchmarks are not new-head evidence.
CUDA/HIP/Metal were not built: this patch touches only the separate SYCL implementation.

Environment, exact build settings and complete launch example

• Ubuntu 24.04, Linux 6.17.0-1007-intel, Intel xe; Xeon w5-3435X, host MemTotal 503.36 GiB.
• One Arc Pro B70, 32 GB VRAM, PCI 0000:cc:00.0; only that card is visible to the test container (logical GPU 0).
• Image intel/omix:0.4.0-devel-ubuntu24.04, image ID sha256:53f0003fa66044935d69065836ba8ac40fab289b94655667fdeac0bdfa7e75c6.
• oneAPI DPC++ 2026.1.1; compute runtime 26.31.39395.13, device driver 1.17.39395+13.
• Container: --memory 34359738368 --memory-swap -1 --cpus 16 --shm-size 1g; one client, no batching. This is a cgroup limit on a larger host, not validation of a physically 32 GiB machine.
• llama.cpp dependency 3cf03257f219afbe7334045ff7c6a06ac68c627d; TLA disabled.
• Qwen3.8-Flash-Next-GSQ-RCO-GGUF, original IQ2_XS, both numbered shards. Model revision ed59f92082b1e93c0e96d60a8b11aab089b52f09.
• MTP from Qwen/Qwen3.8-Flash-Next, revision de4b8e4d43b917e7706784d8bb445c9af86a3540; observed draft vocabulary 106299 (137.9 MiB head).

Build the candidate with the following commands in the oneAPI environment. Build the upstream comparator identically,
substituting /sources/baseline and /task/build-baseline. Sources are pinned to the full commits above.

source /opt/intel/oneapi/setvars.sh
cmake -S /sources/pr2555/sycl -B /task/build-pr2555 -G 'Unix Makefiles' \
  -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx \
  -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
  -DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_SYCL_PARITY=ON \
  -DSTRATA_GGML_DIR=/deps/llama.cpp
cmake --build /task/build-pr2555 --target strata -j 6

The following is a complete candidate configuration sample, not an unexplained environment-variable name.
Save it as /task/pr2555-candidate.json. /sources/baseline is the pinned upstream checkout used for the common Python frontend;
/sources/pr2555 is the candidate checkout; /task is the writable test directory. /opt/strata-venv is the serving Python environment mounted into the container. The /models/Strata/... paths are mounts
for the already-prepared pack/tokenizer, the two original GGUF shards and MTP runtime. Adjust those paths to the local files.

{
  "backend": "sycl",
  "exe": "/task/build-pr2555/strata",
  "cwd": "/sources/baseline",
  "args": [
    "--pack",
    "/models/Strata/ed59f92082b1e93c0e96d60a8b11aab089b52f09/pack",
    "--native",
    "/models/Strata/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf",
    "--ple-gguf",
    "/models/Strata/ed59f92082b1e93c0e96d60a8b11aab089b52f09/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf",
    "--expert-profile",
    "/sources/baseline/data/expert-profile.bin",
    "--expert-cache",
    "auto",
    "--stream-experts",
    "--vram-reserve-mib",
    "1536",
    "--no-prefill-borrow",
    "--prefill",
    "auto",
    "--spec",
    "4",
    "--spec-min-p",
    "0.5",
    "--mtp",
    "/models/Strata/mtp-Qwen3.8-Flash-Next-de4b8e4d43b917e7706784d8bb445c9af86a3540/rt",
    "--max-context",
    "131072",
    "--kv",
    "int8",
    "--adapt-every",
    "8",
    "--prompt-cache",
    "0",
    "--kv-resident",
    "32768",
    "--pcie-frac",
    "0.30",
    "--adapt-decay",
    "1",
    "--suffix-draft",
    "3",
    "--host-core",
    "last"
  ],
  "env": {
    "ONEAPI_DEVICE_SELECTOR": "level_zero:gpu",
    "SYCL_CACHE_PERSISTENT": "0",
    "ZES_ENABLE_SYSMAN": "1",
    "STRATA_VERIFY_DEVICE_PLAN": "1",
    "STRATA_VERIFY_NO_HOST": "1",
    "STRATA_STAGER_THREADS": "12",
    "STRATA_MIRROR_MIB": "16384",
    "STRATA_DECODE_TIMING": "1",
    "ZE_AFFINITY_MASK": "0",
    "STRATA_CACHE_DIR": "/task/shared-calibration",
    "STRATA_SYCL_IQ4_PAIR_REUSE": "1",
    "STRATA_SYCL_IQ4_Q8_SPLIT": "1",
    "STRATA_SYCL_IQ4_LOOKUP": "1"
  },
  "tokenizer": "/models/Strata/ed59f92082b1e93c0e96d60a8b11aab089b52f09/pack/tokenizer",
  "model_name": "qwen3.8-flash-next-iq2_xs",
  "log": "/task/pr2555-candidate.log",
  "host": "127.0.0.1",
  "port": 8080,
  "gpu": 0,
  "open_browser": false,
  "parallel": 1,
  "sampling": {
    "temperature": 0.0,
    "seed": 42,
    "top_p": 1.0,
    "top_k": 64,
    "min_p": 0.0,
    "repetition_penalty": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "penalty_last_n": 64
  }
}

Direct launch with the same frontend/configuration (the collection harness additionally records engine IPC/output identity):

source /opt/intel/oneapi/setvars.sh
cd /sources/baseline
/opt/strata-venv/bin/python -m serve.server --engine strata --config /task/pr2555-candidate.json --port 8080

For the upstream arm, use /task/build-baseline/strata, another log file, and remove only this PR's STRATA_SYCL_* entries.
For the candidate-default correctness arm, keep the candidate executable and remove those same entries.
Leave every common setting, frontend and model unchanged. Clear inherited STRATA_SYCL_*/STRATA_PRIVATE_* variables before launch.
Initialize /task/shared-calibration in an excluded upstream process, issue a full-length warmup, stop it and restart for timing;
all timed processes must read the same unchanged cache rather than calibrating during timing.

Startup checks for the retained run: context 131072, KV int8, resident KV 32768, MTP maximum 4, suffix lookup 3,
spec_min_p=0.50, pool_workers=15, pcie_frac=0.30. INFO spec=6 is the verifier window with MTP maximum 4 and suffix drafting enabled.
Observed initial resident / mirrored expert counts and mirror GiB: [(17756, 6820, 9.17)]. The 16384 MiB setting is the mirror budget, not allocation.

Sampled cgroup memory.current maximum across arms: 1.239–1.260 GiB; sampled swap maximum: 0 bytes. OOM / OOM-kill event deltas: [(0, 0)]. Driver allocations and shared host page cache are not established by these cgroup readings.

Request contents and standalone SSE reproduction example

The small Python example below reconstructs the same request objects used by the retained campaign (21 objects checked against
the three seven-request sequences), and sends one block to the already-running server. Save it as /task/reproduce_requests.py.
It is a portable reproduction client, not the IPC/cgroup collection harness used to produce the table.
The synthetic module may end in a truncated function; that is part of the fixed prompt.

"""Portable request/SSE example; the retained measurement harness is separate."""
import argparse
import hashlib
import json
from pathlib import Path
import time
import urllib.request


def requests(length):
    cap = 128 if length == 4096 else 256
    base = dict(model="strata", temperature=0, reasoning_effort="none",
                max_tokens=cap, stream=True, stream_options=dict(include_usage=True))
    short = dict(base, max_tokens=16, messages=[dict(
        role="user", content="Reply with exactly the word READY.")])
    yield "short-warmup", False, short
    module = "".join(f"def task_{j:05d}(value: int) -> int: return (value * {j % 97 + 1} + {j}) % 100003\n"
                     for j in range(16000))
    suffix = ("\n\nWrite a detailed explanation of the code above. Discuss deterministic integer "
              "transforms, modulo arithmetic, testing, naming, complexity, and maintainability. "
              "Write at least 600 words.")
    sizes = {4096: (7733, 7727), 8192: (15487, 15486), 32768: (62039, 62039)}
    for label, measured, trial in [(f"prompt-warmup-{i}", False, 0) for i in range(1, 4)] + [
            (f"sample-{i}", True, i) for i in range(1, 4)]:
        nonce = (f"trial-{trial}" if trial or length == 32768 else "warmup")
        header = f"Benchmark nonce: series-{length}-{nonce}.\nReview this synthetic Python module:\n"
        content = header + module[:sizes[length][int(bool(trial))]] + suffix
        yield label, measured, dict(base, messages=[dict(role="user", content=content)])


def send(url, body):
    request = urllib.request.Request(url, data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
    start = time.perf_counter_ns()
    first = last = None
    usage = None
    text = []
    with urllib.request.urlopen(request, timeout=1200) as response:
        for line in response:
            if not line.startswith(b"data: ") or line.strip() == b"data: [DONE]":
                continue
            chunk = json.loads(line[6:])
            now = time.perf_counter_ns()
            if chunk.get("error"):
                raise RuntimeError(chunk["error"])
            for choice in chunk.get("choices", []):
                content = choice.get("delta", {}).get("content")
                if content:
                    first = now if first is None else first
                    last = now
                    text.append(content)
            if chunk.get("usage"):
                usage = chunk["usage"]
    assert first is not None and usage is not None
    generated = usage["completion_tokens"]
    assert generated > 1
    return dict(TTFT_s=(first-start)/1e9, TPOT_ms=(last-first)/1e6/(generated-1),
                usage=usage, text="".join(text),
                text_sha256=hashlib.sha256("".join(text).encode()).hexdigest())


if __name__ == "__main__":
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--length", type=int, choices=[4096, 8192, 32768], required=True)
    p.add_argument("--url", default="http://127.0.0.1:8080/v1/chat/completions")
    p.add_argument("--output", type=Path, required=True)
    a = p.parse_args()
    with a.output.open("x") as output:
        for label, measured, body in requests(a.length):
            row = dict(label=label, measured=measured, **send(a.url, body))
            output.write(json.dumps(row)+"\n")
            output.flush()
            print(label, row["TTFT_s"], row["TPOT_ms"], row["usage"], flush=True)

For each of 4096, 8192, 32768, restart the engine for upstream-before, candidate and upstream-after,
and run this client once per arm with a distinct output filename, for example:

/opt/strata-venv/bin/python /task/reproduce_requests.py --length 4096 --output /task/pr2555-4096-before.jsonl

For the 4K default-path check, restart the candidate with all new controls unset and use another output filename; exclude it from timing comparisons.
Exclude the first four requests; compare the final three. For full-length requests, check usage input counts 4096 / 8192 / 32768, actual output counts 128 / 256 / 256,
zero reused prompt tokens and matching output text. The retained campaign additionally checks engine token IDs.
Greedy sampling, seed 42, thinking off, sampler defaults and runtime settings are explicit in the configuration above.
Host page cache is not flushed; no disk-cold TTFT is claimed. Suffix drafting is time-adaptive, so actual work counts are retained below.

Individual measured samples, baseline variation and actual draft work

Every block is one fresh engine, one excluded short warmup plus three excluded full-length warmups, then three measured requests.
Order is upstream-before / enabled candidate / upstream-after. There are six upstream and three candidate samples per length.

Input	Arm	TTFT three samples (s)	TPOT three samples (ms/token)
4096	before	3.147803, 3.151082, 3.151935	14.474981, 14.631803, 15.065819
4096	candidate	3.150134, 3.152997, 3.148791	13.632988, 13.770287, 14.196260
4096	after	3.152742, 3.151423, 3.153340	14.482911, 14.650501, 15.075091
8192	before	6.376100, 6.377828, 6.375774	13.149777, 14.485273, 13.945681
8192	candidate	6.377616, 6.377858, 6.374457	12.397409, 13.660816, 13.159584
8192	after	6.378924, 6.378313, 6.378379	13.146080, 14.481640, 13.945532
32768	before	27.293969, 27.290012, 27.290019	14.939764, 14.614144, 13.625241
32768	candidate	27.299981, 27.281396, 27.288338	14.097065, 13.798316, 12.862501
32768	after	27.333829, 27.283026, 27.290483	14.937587, 14.614376, 13.623334

Input	Before→after upstream drift TTFT / TPOT	Three paired improvements TTFT / TPOT
4096	+0.071% / +0.081%	+0.004%, -0.055%, +0.122% / +5.843%, +5.948%, +5.801%
8192	+0.031% / -0.018%	-0.002%, +0.003%, +0.041% / +5.708%, +5.680%, +5.636%
32768	+0.041% / -0.009%	+0.051%, +0.019%, +0.007% / +5.634%, +5.583%, +5.591%

Input	Arm	Drafts offered / accepted per measured request
4096	before	112 / 73, 117 / 73, 111 / 68
4096	candidate	112 / 73, 117 / 73, 111 / 68
4096	after	112 / 73, 117 / 73, 111 / 68
8192	before	208 / 162, 219 / 148, 206 / 151
8192	candidate	208 / 162, 219 / 148, 206 / 151
8192	after	208 / 162, 219 / 148, 206 / 151
32768	before	218 / 148, 210 / 148, 199 / 156
32768	candidate	218 / 148, 210 / 148, 199 / 156
32768	after	218 / 148, 210 / 148, 199 / 156

Limits

Development comparison on this single B70/model/configuration. Performance is not established for other hardware, workloads or contexts.
Default behavior is preserved by opt-in dispatch; no default change or formal benchmark approval is claimed.
The older measurements below use different source/defaults and RAM limits and are not the current comparison.

Historical pre-beta description and measurements — superseded by the 2026-10-20 retest above

The following original text is retained as historical evidence. In particular its default-on control descriptions and old tables
do not describe the rebased opt-in source or current 32 GiB measurements.

Summary

Reduce IQ4_XS × Q8_1 decode work on Intel Battlemage by sharing loads across adjacent quantization groups and token columns, and replacing per-value codebook lookups with a bit-select tree.

What changed

• sycl/src/kernels/cuda/native_mmvq.dp.cpp: add ESIMD INT8 XMX kernels that reuse IQ4 headers and packed weights across two Q8_1 blocks and up to four token columns. Selected shapes also load the adjacent Q8_1 blocks and their scales together.
• sycl/include/strata/kernels/iq4_lut.hpp: add an exact packed IQ4 bit-select lookup, used by the remaining eligible wide-kernel shapes.
• Dispatch uses upstream intel_gpu_gen / IntelGpuGen::Battlemage. The optimized defaults cover K=2560/6144 and T=1–6, subject to alignment, row-count and shape guards; pair reuse and Q8 splitting cover T≤4. Other devices and unsupported shapes retain the original dispatch. CUDA/HIP implementations and public documentation are unchanged.

Dispatch controls

Control	Unset (default) or 1	0
STRATA_SYCL_IQ4_PAIR_REUSE	Enable pair reuse and allow its Q8-split subpath.	Bypass pair reuse and Q8 splitting; the independent wide-kernel lookup optimization may still run.
Child of pair reuse: STRATA_SYCL_IQ4_Q8_SPLIT	Load adjacent Q8_1 blocks and scales together on selected shapes within the pair-reuse branch.	Use the pair-reuse kernel without Q8 splitting.
STRATA_SYCL_IQ4_LOOKUP	Use the bit-select lookup in eligible wide kernels when the pair-reuse branch is not selected.	Use the original wide-kernel lookup; pair reuse and Q8 splitting are unaffected.

There is no single master switch for this PR. To select the original dispatch, set both STRATA_SYCL_IQ4_PAIR_REUSE=0 and STRATA_SYCL_IQ4_LOOKUP=0. The Q8-split control has no effect when pair reuse is disabled.

All controls retain the device, alignment and shape guards described above. Set them before starting the engine and restart it when changing them: their values are cached on first use. They support diagnosis and fallback; normal operation needs no environment-variable setting.

Validation

One Arc Pro B70 32 GB, Linux/xe, Xeon w5-3435X; intel/omix:0.4.0-devel-ubuntu24.04, oneAPI DPC++ 2026.1.1, compute runtime 26.31.39395.13, Level Zero loader 1.32.0. Container limit: 96 GiB RAM / 16 CPU quota. Builds and GPU tests ran inside this container.

Build:

cmake -S sycl -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx \
  -DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_SYCL_PARITY=ON
cmake --build build --target strata mmvq_bench

At submitted revision e5c0c9c69d38c9469c185ba67c0f7d61e0ecc564:

• IQ4 output comparisons and guard checks pass for default and fallback dispatch. Profiling confirms the expected kernel paths.
• Fresh serving runs with 4096, 8192 and 32768 input tokens each produce 128 output token IDs identical to upstream, with default dispatch and with the optimizations disabled (6 comparisons). These profiled runs are excluded from latency measurements.
• Offline device-image compilation of the affected unit passes for BMG G21, DG2 and PVC; this does not establish runtime correctness or performance on those devices.

Additional checks at measured revision 9683cda6c60139976058d0f2abe90c10c1aff426: ten locally defined requests produce identical upstream token IDs at each max-context setting, 32768 and 131072 (20/20). They cover arithmetic, Python, prose, Chinese, JSON, Unicode and 4096/8192-token synthetic code-review inputs. A frozen 4096-token CLI input with --pcie-frac 0 also produces all 248320 first-token logits byte-identically to upstream, with all values finite.

End-to-end performance

Timing comparison: candidate 9683cda6c601 versus upstream 296b59d2f73c, on the B70 environment described above. The current head (e5c0c9c69d38) has been checked for correctness but has not been remeasured for endpoint latency.

TTFT is the time from the request to the first nonempty SSE token. TPOT is the time between the first and last nonempty SSE tokens divided by generated tokens minus one. Negative changes mean lower latency.

Input / output tokens	TTFT baseline → default (s)	Change	TPOT baseline → default (ms/token)	Change
4096 / 128	3.474050 → 3.474139	+0.003%	14.322062 → 13.491105	-5.802%
8192 / 256	7.027478 → 7.030339	+0.041%	13.606768 → 12.818306	-5.795%
32768 / 256	29.693462 → 29.710838	+0.059%	14.078578 → 13.289964	-5.606%

Test configuration and sampling

Item	Setting
Timing baseline / frontend	296b59d2f73c8f04a805df29c2fa97655adace78
Timed candidate	9683cda6c60139976058d0f2abe90c10c1aff426; this PR only
Model	Qwen3.8-Flash-Next-GSQ-RCO, IQ2_XS; revision ed59f92082b1e93c0e96d60a8b11aab089b52f09
Sampling	Greedy, seed 42, thinking off
KV and context	INT8 KV; max context 131072; resident KV 32768
Expert cache	--expert-cache 17772 experts; reserve 1536 MiB; mirror budget 16384 MiB
Prefill and prompt cache	Auto prefill; no prefill borrowing; prompt cache off
Speculation and adaptation	spec4 / min-p0.5 / suffix3; adapt-every8 / decay1

Observed capacity in every arm: 18609 resident expert slots and 5967 mirrored experts (8.02 GiB). The 16384 MiB setting is a budget, not the actual mirror allocation.

Existing upstream settings held equal in both arms: STRATA_VERIFY_DEVICE_PLAN=1, STRATA_VERIFY_NO_HOST=1, STRATA_STAGER_THREADS=12, STRATA_MIRROR_MIB=16384, STRATA_DECODE_TIMING=1; ONEAPI_DEVICE_SELECTOR=level_zero:gpu, SYCL_CACHE_PERSISTENT=0.

Each arm starts a fresh process. One short request and three full-input requests are excluded as warmups; three requests are then measured.

Input / output tokens	Arm order	Measured requests: baseline / candidate
4096 / 128	Five baseline/candidate pairs, alternating order	15 / 15
8192 / 256	Baseline / candidate / baseline	6 / 3
32768 / 256	Baseline / candidate / baseline	6 / 3

The 4K percentage averages the five pair-mean ratios. The 8K/32K percentage averages the three request-wise ratios against the matching before/after baseline means.

Baseline variability and actual speculative work

4K

Pair changes, TTFT: -0.075%, +0.000%, +0.054%, +0.042%, -0.009%; TPOT: -5.762%, -5.829%, -5.812%, -5.819%, -5.787%.
Baseline first-to-last change: TTFT -0.003%, TPOT +0.009%; block-mean spans 0.081% / 0.052%.
Output token IDs match across all arms, including warmups. Mean windows/offered/accepted per measured request: baseline 54.000/111.000/74.000 (15 requests); candidate 54.000/111.000/74.000 (15 requests).

8K

Baseline drift: TTFT -0.053%, TPOT -0.017%.
Output token IDs match across all arms, including warmups. Mean windows/offered/accepted per measured request: baseline 102.667/209.000/153.667 (6 requests); candidate 102.667/209.000/153.667 (3 requests).

32K

Baseline drift: TTFT +0.009%, TPOT -0.002%.
Output token IDs match across all arms, including warmups. Mean windows/offered/accepted per measured request: baseline 103.333/202.333/154.000 (6 requests); candidate 103.333/201.667/154.000 (3 requests).

Speculation settings are fixed; offered counts include suffix drafts and actual work may differ because suffix selection is time-adaptive. The measurements are warmed and post-readiness; the host page cache was not flushed, so no disk-cold TTFT result is claimed.

Kernel measurements

Independent microbenchmarks use synthetic weights and activations at matrix dimensions used by the model. K and N are the input and output dimensions; T is the tested number of token columns. Timings were collected at 9683cda6c60139976058d0f2abe90c10c1aff426.

The baseline selects the original dispatch in the same binary with the IQ4 optimizations disabled; the candidate uses default dispatch. Each baseline/candidate/baseline arm has three measured samples after a ≥2-second/1000-call warmup; samples last ≥3 seconds/2000 calls. Times include host submission and the final queue wait. These are isolated kernel-call timings; model-level results are reported in the E2E table above.

Operation	K / N / T	Baseline (µs)	Default (µs)	Change	Drift
GDN QKV projection	2560/10240/1	39.476	24.147	-38.831%	+0.006%
GDN QKV projection	2560/10240/2	43.162	25.796	-40.235%	+0.010%
GDN QKV projection	2560/10240/3	47.355	26.392	-44.268%	-0.005%
GDN QKV projection	2560/10240/4	52.056	26.666	-48.774%	+0.007%
GDN QKV projection	2560/10240/5	57.041	49.766	-12.754%	-0.005%
GDN QKV projection	2560/10240/6	65.613	60.329	-8.053%	+0.006%
GDN output projection	6144/2560/1	26.958	19.187	-28.826%	+0.002%
GDN output projection	6144/2560/3	30.833	19.917	-35.404%	-0.007%
GDN output projection	6144/2560/4	34.426	20.518	-40.401%	-0.008%
GDN output projection	6144/2560/6	50.411	45.301	-10.136%	-0.020%
MTP draft head	2560/106299/1	420.032	272.352	-35.159%	+0.164%
Shared expert gate/up	2560/640/3	5.502	3.662	-33.438%	-0.043%

Limitations

Runtime and performance validation is limited to B70. Other GPUs, Windows and multi-GPU operation were not runtime-tested. The patch changes only the two SYCL implementation files listed above; no test-source changes are included.

[deriknel]

Tested on two Arc Pro B70s: layer split 26, --pipeline-windows 2, Swift 1.5 Flash-Next IQ3_S with every expert resident, INT8 KV with 32768 cells resident, engine 296b59d plus our patches (#2433, #2436, #2437). Each arm on a fresh engine; this PR alone against all three of your PRs (#2582, #2555, #2584) switched off, the off arm run twice.

• Greedy text identical on 5 prompts.
• Decode 84.5 / 123.8 / 94.2 tok/s against 83.7 / 122.2-122.8 / 93.5-93.6: +0.7 to +1.3%. This IQ3_S has IQ4_XS only on some dense tensors (the shared expert's gate/up on 13-15 of 48 layers and a few attention projections), so less of the model takes your path than in the IQ2_XS file.
• Prompt reading unchanged: 1,065 / 1,837 / 1,793 tok/s.

We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[xiangyuT]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.