Repository navigation
SYCL port on Windows/OpenCL: 0.1.40-sycl decodes at 22.3 tok/s with --spec 2 where 0.1.39-sycl does 53.6 (and short runs vary 40%) #1397
Description
Summary
On an Arc Pro B70 (32 GB) with the SYCL port on Windows 10 / OpenCL - a backend without command graphs, since the
Windows driver exposes no user-mode Level Zero adapter - engine 0.1.40-sycl decodes 22.3 tok/s with --spec 2
where 0.1.39-sycl decodes 53.6 tok/s on the same card, same flags, interleaved runs.
The same A/B with the MTP draft layer (--spec 4 --spec-min-p 0.5 --mtp) is the other way round: 77.3 against 69.7
tok/s. So this is not "0.1.40 is slower on this backend" - it is specifically the no-draft-layer path.
Numbers
Coder IQ1_M, 12,288/12,288 experts resident, --stream-experts, INT8 KV, greedy 256 tokens, run interleaved
(v1, v2, v1, v2, v1, v2) from the command line, STRATA_VERIFY_EAGER=1 on both:
| run | 0.1.39-sycl | 0.1.40-sycl |
|---|---|---|
--spec 4 --mtp |
69.86, 69.64, 69.69 | 77.30, 77.29, 77.24 |
--spec 2 |
53.64, 53.90, 53.38 | 21.98, 22.85, 22.32 |
Three pairs each, tight spread, and the --spec 2 gap is 2.4x. Same machine, same model, same flags, the only
difference the engine build.
A measurement caveat, which may matter more than the regression
Short generations are not comparable on this backend. The same build and the same flags gave 48.9 and 67.5 tok/s
on two consecutive 128-token runs, and a 155-token answer through the API was ~25% off a 256-token run of the same
build. I read a regression off those short runs first and it did not survive interleaved medians over 256 tokens.
If the numbers in docs/INTEL.md come from runs of that length, they carry that variance. Median of 3+, interleaved
against the other build, would be a safer way to state them.
Not attributed
I have not worked out which part of 0.1.40 makes --spec 2 slower: the drafter-free window, or the suffix drafter's
own cost per round. STRATA_VERIFY_PROFILE=1 on 0.1.40-sycl shows a 4-token window at ~65 ms with ~24 ms of it in the
GDN hyper-connection read, nearly flat from T=2 (23.6 ms) to T=6 (25.6 ms) - a per-layer fixed ~0.5 ms over 48 layers,
which looks like dispatch rather than arithmetic. Whether that differs on the 0.1.39 build I did not measure.
Setup
Windows 10, driver 32.0.101.8976, i9-9900 / 64 GB, conda-forge dpcpp_win-64 2026.1.1, OpenCL backend
(sycl-ls lists only opencl:gpu), --spec / --mtp / --prefill 4096 / --vram-reserve-mib 2048 /
--kv-resident 32768 / --max-context 131072.
Activity
demetree commentedon Oct 7, 2026
A second regression, 0.1.40.3 against 0.1.40.2
Same card, same model, same flags, interleaved runs of the two engines (the 0.1.40.2 build is this PR's branch before
the 0.1.40.3 rebase), greedy,
--spec 4 --mtp:Two pairs each, spread under 0.5 tok/s, same machine back to back. It is not context-length dependent (both rows
lose the same third) and not the draft layer's acceptance.
What I ruled out, and one instrument warning
STRATA_VERIFY_PROFILE=1underSTRATA_VERIFY_EAGER=1is not comparable across builds. Its per-stage numberscome from a host
cs->wait()at every stamp, and on the OpenCL backend that wait is itself expensive, so the splitbetween stages moves with the number of launches. It put one stage (the GDN hyper-connection read) at 4.1 ms on
0.1.39 and 23.9 ms on 0.1.40.2 with identical kernels -
sycl/src/kernels/cuda/fused_gr.dp.cppis byte-identicalbetween those two bases apart from a
boolreturn - which is what sent me looking for a kernel change that is notthere.
STRATA_DECODE_TIMING=1(ms/window, no per-stage syncs) is stable to ±0.3% across runs and is what I woulduse next.
Two more measurement traps on this backend, in case they cost anyone else time
its
STATE_HASHdiffers between two runs. So a greedy output comparison is not a correctness check here.temperature: 0.0request with its own Chat defaults (temperature 1.0, top_p 0.95), so a"greedy" run through it is sampled. That is also why the serve-path and command-line numbers disagree.
The prompt path, for contrast
The same A/B on the prompt: 0.1.40.3's engine with the plain-SYCL GEMM fallback reads a 2,048-token prompt at
14.5 tok/s; with
fallback_gemm_tiled(the one change in this PR) the same binary reads it at 73.5. Measured in onebinary with
STRATA_FALLBACK_NAIVE=1flipping the two kernels, so the 5x is the GEMM and nothing else. The tiledkernel is bit-for-bit the same arithmetic -
STRATA_FALLBACK_SELFTEST=1runs both over a problem that crosses everytile edge and prints the difference: 0 of 4,189 values.
demetree commentedon Oct 8, 2026
Follow-up: it reproduces, it is not the spin bound, and it looks like the new chained-draft policy
Three things, all on an Arc Pro B70 (OpenCL backend, no command graphs), Coder IQ1_M,
--spec 2, 256 greedy tokens.1. It reproduces. Three interleaved pairs, same harness as before, medians of three:
so the -58% stands. (One of my own intermediate passes disagreed with this; it ran the builds in sequence rather than
interleaved and should be ignored - the interleaved medians are the reliable ones.)
2. It is not the spin bound. Both builds have
kSpinMax = 20000(themaxfridbe"experiment: 100x smaller" commitis in v0.1.39 and v0.1.40 alike), and the SYCL sources differ between the two tags by 4 lines - one
NEVER-RANGcudaStreamQuerytranslation fix insycl/src/core/verify.cpp. Nothing else. So this is not the same cause as the0.1.40.3 regression in #1473.
3. What changed is how much the drafter asks for. The engine's own speculation lines, same runs:
0.1.40 drafts T3 half the time where 0.1.39 drafted T2 essentially always, and gets fewer tokens out of it. A
longer window that costs more and returns less is what a mis-priced window looks like, and
src/spec/draft_policy.cppgrew exactly that in 0.1.40:
DraftPolicy::chain()prices a chained draft aschain_rate(match) / cost_ms(t_mtp + k)against
e0 / cost_ms(t_mtp), witht_mtpclamped up to 1 when there is no MTP window to chain onto, and a deliberatelypessimistic prior for short matches (
kChainPrior = {0.45, 0.65, 0.8, 0.9}).It is specific to the no-MTP path in the numbers: the
--spec 4 --mtppath improved over the same span (69.7 -> 77.3tok/s), which is consistent with chaining paying when there is a real MTP window to extend and not paying when there is
not.
cost_ms()is fed measured round times, and on a backend without command graphs those include the per-windowhandshake, so the curve it learns is not the curve on an RTX 5070 that
kShapewas calibrated on.The test I would want run, one line: make
DraftPolicy::chain()return 0 and re-measure--spec 2on the samemachine. If it comes back near 53 tok/s, the cause is the chained policy's sizing and not the kernels. I have not built
that variant, so this is a hypothesis with the evidence above, not an attribution.
Practical note for anyone hitting this on a backend without graphs
-DSTRATA_SYCL_SPIN_MAX=2000000(see #1473) is not only harmless here, it is better than either of the old builds:0.1.40.3 at the long bound does
--spec 2at 60.4 tok/s with 3.29 tokens per round and 179 of 182 suffix draftsaccepted, against 53.6 (0.1.39) and 22.6 (0.1.40) at their own defaults. Same tree, one
-Dapart.SYCL doorbell spin bound chosen per device at run time (#1397)
Niko1221 commentedon Oct 8, 2026
We could not reproduce the first drop on a Linux B70 (Level Zero, with command graphs) with the current engine:
--spec 2(IQ3_S, experts partly resident) decodes 18.2-18.7 tok/s whatever the window: window 4 with the suffix drafter 18.2/18.5/18.4,--suffix-draft 0(window 2) 18.5/18.3/18.3, 3 interleaved pairs. So here it is a fixed cost per round (~54 ms), not the window size.STRATA_VERIFY_EAGER=1against graphs: 18.4/18.6/18.6 vs 18.2/18.7/18.7 (3 pairs). No eager penalty there, so the 2.4x looks specific to the OpenCL adapter, which has no graphs.suffix_draft = 3and the +2 window are the same in both tags. We could not run a 0.1.39 engine on the B70 (it faulted the GPU on its first run), so we stopped there.For the second drop (0.1.40.2 to 0.1.40.3, -37%): the long spin bound (2,000,000 reads) is what 0.1.40.3 gives any build that is not a bmg AOT build. On a Linux B70 built without AOT,
--spec 2, 3 interleaved pairs of the old bound against 20,000: 19.4 against 19.5 tok/s, so on the xe driver the bound does not move decode, and your numbers say the long bound is faster on the OpenCL backend. We therefore do not think the bound explains the -37%. In 0.1.41 it is chosen at run time (20,000 under the xe driver, 2,000,000 elsewhere,STRATA_SPIN_MAX=<reads>overrides), but we are not claiming that as the fix. The other SYCL changes between those tags are small (the A-series wait fix, setup wording, build.sh).Three runs would pin it down: the
--suffix-draft 0A/B on OpenCL (does the window size matter there?), 0.1.40.3 withSTRATA_SPIN_MAX=20000and=2000000once each withSTRATA_DECODE_TIMING=1, andSTRATA_VERIFY_PROFILE=1on 0.1.40.2 and 0.1.40.3 to see which phase grew.demetree commentedon Oct 8, 2026
The bound fix landed, and it needs the drafting configuration, not just the device
sycl/include/strata/sycl_doorbell.hppnow chooses the bound per device at run time - 20,000 reads on a card under thexedriver, 2,000,000 everywhere else, "Windows included (a JIT/OpenCL B-series build measured faster with the long bound, #1397)". That reading is the--spec 2half of this issue. The other half ---spec 4 --mtp, which is whatsetupwrites - goes the other way, and on Windows it now loses most of decode.intel_gpu_driver()reads/sys/class/drm, so on Windows it returns""and every card there gets the long bound whatever the drafting configuration. Re-measured today against currentmain(rebased, so this is the run-time per-device choice and not a build option), Arc Pro B70, OpenCL backend, Coder IQ1_M,--spec 4 --mtp, 256 greedy tokens after a 2,049-token prompt, the same binary oneSTRATA_SPIN_MAXapart, medians of three interleaved runs:STRATA_SPIN_MAX=20000The
--spec 2half is the mirror image, measured on 0.1.40.3: 68.8 tok/s at 2,000,000 against 21.8 at 20,000, because there the suffix drafter's own waits run out inside 20,000 and it produces no draft at all - 2.17 against 1.03 tokens per round. Same for--spec 4without--mtp: 69.7 against 18.9.So one per-device constant cannot serve both. On a backend where a kernel's writes to host-mapped memory are not reliably visible while a window runs, the window's per-layer waits are expected to expire and the short bound is right; the drafting waits genuinely need the long one. The choice has to depend on the drafting configuration - or be made per wait-site - and not only on the device.
What #1390 does for now, since the launcher is the place that knows the configuration:
sycl/serve/strata-sycl.batsetsSTRATA_SPIN_MAX=20000when--mtpis in the arguments and the working adapter's name is a B-series Arc, and leaves the default alone otherwise. An A-series card keeps the long bound even with--mtp, because there the GPU genuinely waits on the host's per-layer CPU expert work and a short bound made it go on with the experts' outputs missing. ASTRATA_SPIN_MAXalready in the environment wins, and-DSTRATA_SYCL_SPIN_MAXstill fixes one bound for a whole build.Not tested: an A-series card on Windows, and Level Zero - this driver still exposes no user-mode adapter, which is a separate thread on #1390.
sycl: spin bound for a B-series card found from the device (Windows/O…
Niko1221 commentedon Oct 10, 2026
Thank you for the follow-ups; the interleaved medians and the note that 0.1.39 and 0.1.40 both have the 20,000 bound made this clear. The spin bound is not the cause of the
--spec 2gap (as you found), so this report's remaining cause is the chained-draft policy (T3 windows half the time, 1.08 tokens a round), which we have not changed in v0.1.42. What did change: a B-series card on Windows / OpenCL now gets the right spin bound for each mode (see #1473), including the long bound for a run that drafts without--mtp. If you still see the 22 tok/s--spec 2result on v0.1.42, please send thespeculationline of both builds; we would look at the draft policy next.Niko1221 commentedon Oct 13, 2026
Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.
If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.
SYCL doorbell spin bound chosen per device at run time (Niko1221#1397)
sycl: spin bound for a B-series card found from the device (Windows/O…