Skip to content

[Engine] [SYCL] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit) #3150

Open
Open
[Engine] [SYCL] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)#3150

Description

@xiangyuT
xiangyuT
opened on Oct 19, 2026
  • Problem: The unmodified 0.2.0-beta SYCL engine has measurable single-B70 IQ2_XS inference latency that the combined engine preview reduces. For the measured fresh 32K workload at a 128 GiB container RAM limit, client decode rate is 67.10 tok/s upstream versus 94.85 tok/s with the preview.
  • Environment: Strata 0.2.0-beta / engine 0.1.45.3; pure upstream 3d1fb48090e8f2cefc22d7b5845fb7f5b71f33fa, measured preview c3ed1d047768d94f43d7384ccf740a214f5a53dd. Published PR head cb436853ab2ee238fa579a3a4f42a0412b46c439 has identical engine/server source to that measured preview. Ubuntu 24.04.2 LTS, kernel 6.17.0-1007-intel; one Intel Arc Pro B70, 32 GB VRAM; Xeon w5-3435X, 16 CPU quota / 15 workers. Host RAM is 503.36 GiB usable; the example uses a 128 GiB container limit. Original Qwen3.8-Flash-Next GSQ/RCO IQ2_XS, quantization revision ed59f92082b1e93c0e96d60a8b11aab089b52f09; oneAPI 2026.1 / AOT bmg-g31; INT8 KV, 32K resident window, max context 131584, spec4/min-p0.5/suffix3, thinking off and vision disabled.
  • Reproduce: Build the pinned upstream and preview SYCL sources using the current build/configuration and replay instructions below and in draft PR #3153. Start each arm through the single-client native-GGUF serving/SSE path, with --prefill auto --prompt-cache 0 --spec 4 --spec-min-p 0.5 --suffix-draft 3 --max-context 131584 --kv int8 --kv-resident 32768 --pcie-frac 0.30; for this B70 example upstream uses --expert-cache auto, preview --expert-cache 16000 (B60 uses auto on both). Keep the documented model/profile, sampler and environment controls, including upstream's default ESIMD INT8 auto path. Execute one short warmup and three fresh full-length warmups, followed by three fresh 32,768-input-token requests with a 256-token output cap, greedy/seed42 and no prompt-prefix reuse. Compare the same request sequence across arms using the client TTFT/TPOT definitions below. Full independent reproduction is pending publication of the sanitized prompt corpus and measurement harness.
  • Expected: Preserve generated text and token IDs for the fixed comparison inputs while reducing serving latency with the selected engine optimizations; retain upstream behavior outside their supported scope. Default-path byte identity remains a separate, uncompleted validation requirement.
  • Actual: At 32K/128 GiB, upstream versus preview TTFT is 27.390 versus 22.883 seconds and client TPOT is 14.903 versus 10.543 ms (29.26% reduction). All seven paired request outputs/token-ID sequences for that cell, including warmups, match. The 1K TTFT tradeoff is retained: 0.962 versus 1.036 seconds (7.68% regression). No engine crash or runtime error was observed in these completed comparison cells.
  • Evidence: The expandable evidence block immediately below the current tables contains sanitized per-request timing/count fields for 156 measured requests (96 B70 and 60 B60), median/range summaries, actual startup log excerpts confirming the upstream ESIMD default and adaptation behavior, and the output of nine passing SYCL regression cases. Full sanitized prompt/output/log artifacts have not yet been attached.
  • Reproduction status: Reproduced locally on the pinned upstream and measured preview revisions above. This is measured engine behavior, with the completed B70/B60 matrices and remaining validation limits stated below.

Current beta E2E comparison (2026-10-20)

B70 collection is complete: 32/64/128 GiB × five prompt lengths, 15 upstream/preview pairs, plus two 32K stability blocks; 96 measured requests in 32 blocks. All paired text and token-ID sequences match, including warmups. The 1K TTFT regression is retained. B60 collection is also complete: 128/32 GiB × five lengths, 10 pairs / 60 measured requests. Combined total: 25 paired configurations and 156 measured requests across 52 blocks.

Baseline: unmodified 0.2.0-beta / engine 0.1.45.3, 3d1fb48090e8f2cefc22d7b5845fb7f5b71f33fa. Measured preview: c3ed1d047768d94f43d7384ccf740a214f5a53dd; published PR head cb436853ab2ee238fa579a3a4f42a0412b46c439 has identical engine/server source. Main observed during this retest (e88ee39d1d6be6e3364183af3b39b62a7ed6f211) has identical baseline engine/server source; it was not separately rebuilt.

One B70, 32 GB VRAM, 16 CPU quota / 15 workers, fresh prompts, thinking off, greedy/seed42, spec4/min-p0.5/suffix3, 256 output tokens. Each cell is the mean of three measured requests, excluding one short and three full-length warmups. TTFT is dispatch to first nonempty answer delta; TPOT is first-to-last nonempty answer-delta interval divided by 255. Client tok/s = 1000 / mean(TPOT ms), distinct from engine decode throughput. Reductions use unrounded means; negative means a regression. Actual speculative work is recorded rather than forced equal.

B70, 128 GiB container RAM limit — complete five-length comparison

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Upstream client tok/s Preview client tok/s
1K 0.962 1.036 -7.68% 14.057 9.777 30.45% 71.14 102.28
4K 3.161 2.810 11.10% 13.696 9.353 31.71% 73.01 106.92
8K 6.437 5.519 14.26% 14.413 10.051 30.26% 69.38 99.49
32K 27.390 22.883 16.45% 14.903 10.543 29.26% 67.10 94.85
128K 129.568 109.751 15.29% 14.405 10.329 28.30% 69.42 96.82

B70, 64 GiB container RAM limit — complete five-length comparison

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Upstream client tok/s Preview client tok/s
1K 0.965 1.036 -7.32% 14.059 9.778 30.45% 71.13 102.27
4K 3.162 2.811 11.12% 13.696 9.353 31.71% 73.02 106.91
8K 6.439 5.520 14.27% 14.413 10.055 30.24% 69.38 99.46
32K 27.388 22.885 16.44% 14.906 10.544 29.26% 67.09 94.84
128K 129.545 109.715 15.31% 14.402 10.319 28.35% 69.43 96.91

B70, 32 GiB container RAM limit — complete five-length comparison

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Upstream client tok/s Preview client tok/s
1K 0.965 1.038 -7.57% 14.060 9.778 30.46% 71.12 102.27
4K 3.160 2.810 11.09% 13.708 9.348 31.80% 72.95 106.97
8K 6.439 5.521 14.26% 14.414 10.049 30.28% 69.38 99.52
32K 27.384 22.883 16.44% 14.903 10.542 29.26% 67.10 94.86
128K 129.550 109.745 15.29% 14.408 10.328 28.32% 69.41 96.83

At 32K/128 GiB, baseline-after and an independent preview repeat also passed: baseline TPOT drift -0.0048%, TTFT drift -0.0083%; all three paired samples improved for both latency metrics in both preview blocks. Preview repeats measured 94.85 and 94.86 client tok/s.

B60, 128 GiB container RAM limit — complete five-length comparison

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Upstream client tok/s Preview client tok/s
1K 1.362 1.527 -12.12% 22.554 13.915 38.31% 44.34 71.87
4K 4.719 4.500 4.64% 22.352 13.690 38.75% 44.74 73.05
8K 9.784 8.962 8.40% 23.890 14.381 39.80% 41.86 69.54
32K 42.165 37.731 10.51% 22.857 14.028 38.63% 43.75 71.29
128K 203.410 184.296 9.40% 23.809 14.871 37.54% 42.00 67.25

B60, 32 GiB container RAM limit — complete five-length comparison

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Upstream client tok/s Preview client tok/s
1K 1.363 1.525 -11.87% 22.556 13.900 38.38% 44.33 71.94
4K 4.721 4.501 4.67% 22.351 13.726 38.59% 44.74 72.86
8K 9.780 8.965 8.34% 23.891 14.380 39.81% 41.86 69.54
32K 42.172 37.751 10.48% 22.864 14.054 38.53% 43.74 71.16
128K 203.419 184.289 9.40% 23.813 14.872 37.54% 41.99 67.24

B60 uses one 24 GB Arc Pro B60, AOT bmg-g21, TLA off, Xeon w9-3595X; a subsequent read-only host capture found 16×32 GiB DDR5, 8 populated channels / 2 DIMMs per channel, SMBIOS configured 4400 MT/s (DIMM rating 4800 MT/s). This hardware capture was made after the benchmark. B60 retains a 16-CPU quota / 15 workers, --expert-cache auto on both arms, and a 32 GiB host-mirror budget. B70 uses the configuration above, including preview expert-cache 16000; these are per-host engine comparisons, not an isolated GPU comparison. All ten B60 paired text and token-ID sequences match, including warmups. The B60 1K TTFT regressions are retained: +12.12% at 128 GiB and +11.87% at 32 GiB. Both test containers were removed after exporting evidence.

All 52 beta block startup logs (B70: 32; B60: 20) report strata mtp: draft head over 106299 tokens (137.9 MiB). Both baseline and preview therefore used the standard draft-vocabulary subset; these results do not measure the full 248,320-token head or the 40,525-token English subset.

RAM tiers above are container limits in GiB with shared model page cache retained; they do not establish minimum installed RAM. This is custom development characterization, not formal acceptance or an approved standard benchmark.

All 96 measured requests, median/range summaries and validation excerpts

Three samples per block; warmups are excluded from summaries. Engine decode rates here are the engine's reported rates, separate from client TPOT-based tok/s.

Block TTFT s median [min, max] TPOT ms median [min, max] Engine decode tok/s median [min, max]
ram128-preview-1024 1.0363 [1.0349, 1.0367] 9.8711 [9.5729, 9.8876] 101.0000 [100.8000, 104.1000]
ram128-preview-131072 109.7569 [109.7193, 109.7754] 10.1098 [9.9999, 10.8759] 98.5000 [91.6000, 99.6000]
ram128-preview-32768 22.8815 [22.8697, 22.8982] 10.5104 [10.3924, 10.7264] 94.9000 [93.0000, 95.9000]
ram128-preview-4096 2.8101 [2.8078, 2.8118] 9.2939 [9.2339, 9.5311] 107.2000 [104.5000, 107.9000]
ram128-preview-8192 5.5175 [5.5131, 5.5262] 10.0714 [9.8855, 10.1964] 99.0000 [97.8000, 100.8000]
ram128-preview-repeat-32768 22.8853 [22.8641, 22.9032] 10.5086 [10.3934, 10.7244] 94.9000 [93.0000, 95.9000]
ram128-upstream-1024 0.9618 [0.9610, 0.9634] 14.0318 [13.9007, 14.2384] 71.1000 [70.1000, 71.7000]
ram128-upstream-131072 129.5678 [129.5573, 129.5775] 14.2146 [14.0829, 14.9170] 70.1000 [66.9000, 70.8000]
ram128-upstream-32768 27.3930 [27.3717, 27.4046] 14.8334 [14.8115, 15.0640] 67.2000 [66.2000, 67.3000]
ram128-upstream-4096 3.1611 [3.1580, 3.1635] 13.8315 [13.3663, 13.8902] 72.1000 [71.8000, 74.6000]
ram128-upstream-8192 6.4365 [6.4365, 6.4378] 14.5439 [13.9056, 14.7883] 68.6000 [67.5000, 71.7000]
ram128-upstream-after-32768 27.3847 [27.3724, 27.4054] 14.8302 [14.8118, 15.0648] 67.3000 [66.2000, 67.3000]
ram32-preview-1024 1.0380 [1.0359, 1.0388] 9.8659 [9.5811, 9.8873] 101.1000 [100.8000, 104.0000]
ram32-preview-131072 109.7479 [109.7030, 109.7845] 10.1095 [9.9947, 10.8788] 98.5000 [91.6000, 99.7000]
ram32-preview-32768 22.8840 [22.8636, 22.9017] 10.5074 [10.3941, 10.7250] 94.9000 [93.0000, 95.9000]
ram32-preview-4096 2.8093 [2.8084, 2.8119] 9.2868 [9.2325, 9.5249] 107.3000 [104.6000, 107.9000]
ram32-preview-8192 5.5224 [5.5148, 5.5244] 10.0708 [9.8833, 10.1920] 99.0000 [97.8000, 100.8000]
ram32-upstream-1024 0.9641 [0.9631, 0.9666] 14.0352 [13.9043, 14.2415] 71.1000 [70.0000, 71.7000]
ram32-upstream-131072 129.5551 [129.5366, 129.5586] 14.2165 [14.0891, 14.9182] 70.1000 [66.9000, 70.8000]
ram32-upstream-32768 27.3844 [27.3659, 27.4006] 14.8328 [14.8107, 15.0660] 67.3000 [66.2000, 67.4000]
ram32-upstream-4096 3.1608 [3.1593, 3.1614] 13.8448 [13.3807, 13.8981] 72.0000 [71.8000, 74.5000]
ram32-upstream-8192 6.4370 [6.4368, 6.4423] 14.5467 [13.9090, 14.7851] 68.6000 [67.5000, 71.7000]
ram64-preview-1024 1.0358 [1.0348, 1.0366] 9.8672 [9.5781, 9.8900] 101.0000 [100.8000, 104.1000]
ram64-preview-131072 109.7252 [109.6862, 109.7339] 10.1052 [9.9826, 10.8692] 98.5000 [91.7000, 99.8000]
ram64-preview-32768 22.8846 [22.8637, 22.9076] 10.5078 [10.4022, 10.7232] 94.9000 [93.0000, 95.8000]
ram64-preview-4096 2.8106 [2.8078, 2.8133] 9.2885 [9.2339, 9.5374] 107.3000 [104.5000, 107.9000]
ram64-preview-8192 5.5206 [5.5166, 5.5231] 10.0813 [9.8839, 10.1985] 98.9000 [97.7000, 100.8000]
ram64-upstream-1024 0.9643 [0.9642, 0.9666] 14.0350 [13.9029, 14.2402] 71.1000 [70.0000, 71.7000]
ram64-upstream-131072 129.5441 [129.5373, 129.5542] 14.2153 [14.0784, 14.9130] 70.1000 [66.9000, 70.8000]
ram64-upstream-32768 27.3930 [27.3659, 27.4051] 14.8409 [14.8109, 15.0665] 67.2000 [66.2000, 67.4000]
ram64-upstream-4096 3.1601 [3.1598, 3.1662] 13.8353 [13.3649, 13.8868] 72.1000 [71.8000, 74.6000]
ram64-upstream-8192 6.4397 [6.4370, 6.4408] 14.5473 [13.9037, 14.7874] 68.6000 [67.5000, 71.7000]

Sanitized per-request measurement fields (CSV):

block,sample,TTFT_s,TPOT_ms,prompt_tokens,reused,output_tokens,engine_prompt_ms,engine_decode_ms,engine_decode_tok_s,drafts_offered,drafts_accepted,finish
ram128-preview-1024,sample-1,1.0366879049688578,9.88760947195046,1024,0,256,1015.9,2539.0,100.8,213,150,length
ram128-preview-1024,sample-2,1.034909341018647,9.871070169131546,1024,0,256,1014.4,2534.4,101.0,218,148,length
ram128-preview-1024,sample-3,1.0363243208266795,9.572916262436147,1024,0,256,1015.7,2458.8,104.1,214,149,length
ram128-preview-131072,sample-1,109.75690422393382,10.875885973812318,131072,0,256,109674.9,2793.7,91.6,211,152,length
ram128-preview-131072,sample-2,109.77536785695702,10.109848003177081,131072,0,256,109692.8,2599.4,98.5,205,159,length
ram128-preview-131072,sample-3,109.71930587710813,9.999861164639393,131072,0,256,109636.1,2569.7,99.6,201,158,length
ram128-preview-32768,sample-1,22.89824545988813,10.510421867537147,32768,0,256,22861.7,2698.9,94.9,215,152,length
ram128-preview-32768,sample-2,22.869661701843143,10.726393678901243,32768,0,256,22833.3,2754.0,93.0,224,145,length
ram128-preview-32768,sample-3,22.881505776196718,10.392421584430279,32768,0,256,22843.7,2669.0,95.9,198,149,length
ram128-preview-4096,sample-1,2.811849548947066,9.531136399463696,4096,0,256,2788.4,2449.0,104.5,222,158,length
ram128-preview-4096,sample-2,2.807837171945721,9.233945113259788,4096,0,256,2784.8,2373.0,107.9,207,161,length
ram128-preview-4096,sample-3,2.8101384327746928,9.293884595892592,4096,0,256,2786.8,2387.4,107.2,215,158,length
ram128-preview-8192,sample-1,5.526170247234404,9.885491949378276,8192,0,256,5501.3,2539.2,100.8,214,156,length
ram128-preview-8192,sample-2,5.517513420898467,10.07142873006124,8192,0,256,5492.2,2586.9,99.0,225,149,length
ram128-preview-8192,sample-3,5.513110249768943,10.19635333807445,8192,0,256,5488.0,2618.7,97.8,220,145,length
ram128-preview-repeat-32768,sample-1,22.903241822961718,10.508589525067924,32768,0,256,22866.2,2698.6,94.9,215,152,length
ram128-preview-repeat-32768,sample-2,22.86406627204269,10.724358678813658,32768,0,256,22827.2,2753.6,93.0,224,145,length
ram128-preview-repeat-32768,sample-3,22.88531443895772,10.393375552752438,32768,0,256,22847.4,2669.0,95.9,198,149,length
ram128-upstream-1024,sample-1,0.961805856321007,13.900676371493176,1024,0,256,935.3,3568.1,71.7,213,150,length
ram128-upstream-1024,sample-2,0.9609621502459049,14.238360465741625,1024,0,256,934.8,3653.9,70.1,218,148,length
ram128-upstream-1024,sample-3,0.9634076119400561,14.031795357518336,1024,0,256,937.2,3601.3,71.1,214,149,length
ram128-upstream-131072,sample-1,129.55730569874868,14.917031385223655,131072,0,256,129468.3,3829.2,66.9,211,152,length
ram128-upstream-131072,sample-2,129.5774956010282,14.082883630751395,131072,0,256,129488.5,3616.6,70.8,205,159,length
ram128-upstream-131072,sample-3,129.56775243626907,14.21464012168786,131072,0,256,129479.0,3650.0,70.1,201,158,length
ram128-upstream-32768,sample-1,27.39300425397232,14.811520897509421,32768,0,256,27349.8,3801.4,67.3,216,151,length
ram128-upstream-32768,sample-2,27.371654233895242,15.063957663654696,32768,0,256,27328.4,3865.4,66.2,223,146,length
ram128-upstream-32768,sample-3,27.404567080084234,14.83343545091795,32768,0,256,27361.2,3806.9,67.2,198,149,length
ram128-upstream-4096,sample-1,3.1635436369106174,13.831501224023455,4096,0,256,3134.8,3551.3,72.1,222,158,length
ram128-upstream-4096,sample-2,3.158033211249858,13.366339543797807,4096,0,256,3129.4,3432.2,74.6,207,161,length
ram128-upstream-4096,sample-3,3.1610699906013906,13.890210035092691,4096,0,256,3132.7,3564.9,71.8,214,157,length
ram128-upstream-8192,sample-1,6.437766758725047,13.905617706112418,8192,0,256,6407.2,3570.0,71.7,213,156,length
ram128-upstream-8192,sample-2,6.436535275075585,14.543913335849842,8192,0,256,6405.8,3732.6,68.6,225,149,length
ram128-upstream-8192,sample-3,6.436532443854958,14.788347212415115,8192,0,256,6405.9,3795.2,67.5,218,145,length
ram128-upstream-after-32768,sample-1,27.384688698220998,14.811771437890974,32768,0,256,27341.0,3801.2,67.3,216,151,length
ram128-upstream-after-32768,sample-2,27.372400103136897,15.064767108974504,32768,0,256,27329.1,3865.9,66.2,223,146,length
ram128-upstream-after-32768,sample-3,27.405355839058757,14.830228399631439,32768,0,256,27362.1,3806.0,67.3,198,149,length
ram32-preview-1024,sample-1,1.038795046042651,9.887346747678285,1024,0,256,1018.0,2538.9,100.8,213,150,length
ram32-preview-1024,sample-2,1.037998863030225,9.8658825143003,1024,0,256,1017.4,2533.3,101.1,218,148,length
ram32-preview-1024,sample-3,1.0359237329103053,9.58109483962842,1024,0,256,1015.3,2460.5,104.0,214,149,length
ram32-preview-131072,sample-1,109.74785775132477,10.878754916655666,131072,0,256,109665.4,2794.6,91.6,211,152,length
ram32-preview-131072,sample-2,109.78445797320455,10.109505296556973,131072,0,256,109694.0,2599.5,98.5,205,159,length
ram32-preview-131072,sample-3,109.70296505698934,9.994703646310988,131072,0,256,109617.1,2568.6,99.7,201,158,length
ram32-preview-32768,sample-1,22.901670003775507,10.507384435657192,32768,0,256,22863.8,2698.2,94.9,215,152,length
ram32-preview-32768,sample-2,22.863574544899166,10.72499521953218,32768,0,256,22826.3,2753.8,93.0,224,145,length
ram32-preview-32768,sample-3,22.8839897448197,10.394079387918406,32768,0,256,22845.8,2669.4,95.9,198,149,length
ram32-preview-4096,sample-1,2.8119274051859975,9.524922673681788,4096,0,256,2788.6,2447.6,104.6,222,158,length
ram32-preview-4096,sample-2,2.809272525832057,9.232547643649227,4096,0,256,2786.0,2372.9,107.9,207,161,length
ram32-preview-4096,sample-3,2.8083514720201492,9.286761059261421,4096,0,256,2785.0,2385.7,107.3,215,158,length
ram32-preview-8192,sample-1,5.524380489718169,9.883293365219645,8192,0,256,5499.0,2538.8,100.8,214,156,length
ram32-preview-8192,sample-2,5.522440382745117,10.070805404992665,8192,0,256,5497.3,2586.6,99.0,225,149,length
ram32-preview-8192,sample-3,5.514843829907477,10.191958603978742,8192,0,256,5489.7,2617.3,97.8,220,145,length
ram32-upstream-1024,sample-1,0.9630755442194641,13.9042666346273,1024,0,256,936.8,3568.9,71.7,213,150,length
ram32-upstream-1024,sample-2,0.9641351541504264,14.24153121188283,1024,0,256,937.7,3654.9,70.0,218,148,length
ram32-upstream-1024,sample-3,0.9665630939416587,14.035153325062756,1024,0,256,940.1,3602.4,71.1,214,149,length
ram32-upstream-131072,sample-1,129.53662011539564,14.918214586727759,131072,0,256,129440.3,3829.3,66.9,211,152,length
ram32-upstream-131072,sample-2,129.55509566608816,14.089149987215505,131072,0,256,129462.9,3617.9,70.8,205,159,length
ram32-upstream-131072,sample-3,129.55864498671144,14.216533863880471,131072,0,256,129464.7,3650.5,70.1,201,158,length
ram32-upstream-32768,sample-1,27.384366606827825,14.810688859399628,32768,0,256,27341.5,3800.9,67.4,216,151,length
ram32-upstream-32768,sample-2,27.36587103176862,15.065999604834646,32768,0,256,27322.3,3866.1,66.2,223,146,length
ram32-upstream-32768,sample-3,27.400607748888433,14.832777078902604,32768,0,256,27358.0,3806.5,67.3,198,149,length
ram32-upstream-4096,sample-1,3.1608067671768367,13.844773768648214,4096,0,256,3131.9,3554.5,72.0,222,158,length
ram32-upstream-4096,sample-2,3.1592900613322854,13.380700952428228,4096,0,256,3130.5,3436.2,74.5,207,161,length
ram32-upstream-4096,sample-3,3.1613849150016904,13.898095422807861,4096,0,256,3132.6,3566.8,71.8,214,157,length
ram32-upstream-8192,sample-1,6.442331418860704,13.909037102598186,8192,0,256,6411.4,3571.1,71.7,213,156,length
ram32-upstream-8192,sample-2,6.437003660947084,14.54672354510894,8192,0,256,6406.3,3733.5,68.6,225,149,length
ram32-upstream-8192,sample-3,6.436790566891432,14.785073655566164,8192,0,256,6406.2,3794.3,67.5,218,145,length
ram64-preview-1024,sample-1,1.0365623221732676,9.89002959583612,1024,0,256,1015.8,2539.6,100.8,213,150,length
ram64-preview-1024,sample-2,1.0348044708371162,9.867226410949348,1024,0,256,1014.6,2533.5,101.0,218,148,length
ram64-preview-1024,sample-3,1.0358372880145907,9.578132180168348,1024,0,256,1015.4,2460.0,104.1,214,149,length
ram64-preview-131072,sample-1,109.72521255584434,10.86924827347199,131072,0,256,109639.1,2791.8,91.7,211,152,length
ram64-preview-131072,sample-2,109.73391232686117,10.105233581042757,131072,0,256,109649.6,2598.2,98.5,205,159,length
ram64-preview-131072,sample-3,109.68620965909213,9.982564913875917,131072,0,256,109603.0,2565.2,99.8,201,158,length
ram64-preview-32768,sample-1,22.907624046318233,10.507821678823115,32768,0,256,22869.7,2698.5,94.9,215,152,length
ram64-preview-32768,sample-2,22.86374979512766,10.723180807761702,32768,0,256,22826.6,2753.2,93.0,224,145,length
ram64-preview-32768,sample-3,22.88455961411819,10.402197830890323,32768,0,256,22846.7,2671.5,95.8,198,149,length
ram64-preview-4096,sample-1,2.813281220383942,9.537374520418691,4096,0,256,2790.3,2450.6,104.5,222,158,length
ram64-preview-4096,sample-2,2.81062196707353,9.233949490475888,4096,0,256,2787.5,2373.2,107.9,207,161,length
ram64-preview-4096,sample-3,2.8077552542090416,9.288461787589625,4096,0,256,2784.6,2386.0,107.3,215,158,length
ram64-preview-8192,sample-1,5.523070538882166,9.883873458659531,8192,0,256,5497.5,2539.2,100.8,214,156,length
ram64-preview-8192,sample-2,5.520641604904085,10.081257760597795,8192,0,256,5495.5,2589.4,98.9,225,149,length
ram64-preview-8192,sample-3,5.516587184276432,10.198488975783773,8192,0,256,5491.5,2619.0,97.7,220,145,length
ram64-upstream-1024,sample-1,0.9642751938663423,13.902856287199493,1024,0,256,938.3,3568.1,71.7,213,150,length
ram64-upstream-1024,sample-2,0.9642413151450455,14.24017339104823,1024,0,256,937.8,3654.6,70.0,218,148,length
ram64-upstream-1024,sample-3,0.9666406130418181,14.034963583610221,1024,0,256,940.6,3602.1,71.1,214,149,length
ram64-upstream-131072,sample-1,129.53728309692815,14.913037592800809,131072,0,256,129449.2,3828.2,66.9,211,152,length
ram64-upstream-131072,sample-2,129.55419597122818,14.078357430430604,131072,0,256,129461.3,3615.3,70.8,205,159,length
ram64-upstream-131072,sample-3,129.54411434195936,14.215255535992922,131072,0,256,129457.1,3650.4,70.1,201,158,length
ram64-upstream-32768,sample-1,27.393013606313616,14.810930155948096,32768,0,256,27349.0,3801.0,67.4,216,151,length
ram64-upstream-32768,sample-2,27.365869925823063,15.066450847057151,32768,0,256,27322.7,3866.3,66.2,223,146,length
ram64-upstream-32768,sample-3,27.405077799223363,14.840921975087886,32768,0,256,27361.4,3808.7,67.2,198,149,length
ram64-upstream-4096,sample-1,3.159828953910619,13.835296838306913,4096,0,256,3130.6,3552.4,72.1,222,158,length
ram64-upstream-4096,sample-2,3.160058844834566,13.364904952765096,4096,0,256,3131.0,3432.4,74.6,207,161,length
ram64-upstream-4096,sample-3,3.1661593089811504,13.886825039106256,4096,0,256,3137.0,3564.5,71.8,214,157,length
ram64-upstream-8192,sample-1,6.440796766895801,13.903662750460938,8192,0,256,6410.2,3569.4,71.7,213,156,length
ram64-upstream-8192,sample-2,6.439688881859183,14.547319105808057,8192,0,256,6408.6,3733.6,68.6,225,149,length
ram64-upstream-8192,sample-3,6.436999605037272,14.787429956463622,8192,0,256,6406.2,3794.9,67.5,218,145,length

Selected regression output, actually executed at c3ed1d047768d94f43d7384ccf740a214f5a53dd:

Start  1: gemm_stride_parity
1/9 Test  #1: gemm_stride_parity ...............   Passed    0.27 sec
    Start  6: gdn_parity
2/9 Test  #6: gdn_parity .......................   Passed    1.98 sec
    Start 10: quantize_act_parity
3/9 Test #10: quantize_act_parity ..............   Passed    0.13 sec
    Start 16: qsa_parity
4/9 Test #16: qsa_parity .......................   Passed    0.55 sec
    Start 21: prefill_gdn_parity
5/9 Test #21: prefill_gdn_parity ...............   Passed    1.86 sec
    Start 26: verify_parity
6/9 Test #26: verify_parity ....................   Passed    0.15 sec
    Start 27: native_multi_parity
7/9 Test #27: native_multi_parity ..............   Passed    0.19 sec
    Start 30: gr_multi_parity
8/9 Test #30: gr_multi_parity ..................   Passed    0.34 sec
    Start 34: esimd_moe_parity
9/9 Test #34: esimd_moe_parity .................   Passed   18.82 sec

100% tests passed, 0 tests failed out of 9

Total Test time (real) =  24.28 sec

Observed upstream log excerpts at the 1K checkpoint:

strata: int8 prompt experts are on (Battlemage default, auto); STRATA_PREFILL_MMQ=0 turns them off
strata: int8 experts (auto): 498 groups took the int8 chain, 2 the FP16 chain
strata generate: adaptive swaps off: with STRATA_VERIFY_NO_HOST an expert they move out of VRAM would be neither there nor mirrored

Original complete request/stream records and warmups are retained. The complete sanitized prompt corpus, outputs and engine logs are not yet attached publicly; the fields and excerpts here are not a complete independent reproduction package.

All 60 B60 measured requests, median/range summaries and startup evidence

Three measured requests per block; warmups excluded. Client and engine throughput use the distinct definitions above.

B60 block TTFT s median [min, max] TPOT ms median [min, max] Engine decode tok/s median [min, max]
ram128-preview-1024 1.5268 [1.5259, 1.5270] 14.0913 [13.4978, 14.1553] 70.8000 [70.5000, 73.9000]
ram128-preview-131072 184.2930 [184.2746, 184.3190] 14.8120 [14.3243, 15.4756] 67.3000 [64.5000, 69.6000]
ram128-preview-32768 37.7346 [37.7189, 37.7407] 14.1633 [13.5791, 14.3406] 70.4000 [69.5000, 73.2000]
ram128-preview-4096 4.4991 [4.4990, 4.5024] 13.4439 [13.0553, 14.5710] 74.2000 [68.5000, 76.3000]
ram128-preview-8192 8.9624 [8.9617, 8.9628] 14.4111 [14.3076, 14.4230] 69.2000 [69.2000, 69.7000]
ram128-upstream-1024 1.3618 [1.3608, 1.3621] 22.4569 [21.9061, 23.3003] 44.4000 [42.8000, 45.5000]
ram128-upstream-131072 203.4115 [203.3929, 203.4241] 23.5693 [23.3436, 24.5127] 42.3000 [40.7000, 42.7000]
ram128-upstream-32768 42.1622 [42.1565, 42.1754] 22.8809 [22.5211, 23.1696] 43.6000 [43.1000, 44.3000]
ram128-upstream-4096 4.7197 [4.7171, 4.7205] 21.7813 [21.7493, 23.5261] 45.8000 [42.4000, 45.9000]
ram128-upstream-8192 9.7830 [9.7805, 9.7897] 23.7705 [23.4832, 24.4153] 42.0000 [40.9000, 42.5000]
ram32-preview-1024 1.5255 [1.5238, 1.5259] 14.0761 [13.4964, 14.1271] 70.9000 [70.6000, 73.9000]
ram32-preview-131072 184.3009 [184.2369, 184.3306] 14.8071 [14.3149, 15.4952] 67.3000 [64.4000, 69.6000]
ram32-preview-32768 37.7486 [37.7479, 37.7578] 14.1919 [13.6127, 14.3568] 70.3000 [69.5000, 73.1000]
ram32-preview-4096 4.5002 [4.4974, 4.5044] 13.4645 [13.1051, 14.6076] 74.0000 [68.3000, 76.1000]
ram32-preview-8192 8.9644 [8.9599, 8.9694] 14.4063 [14.3067, 14.4284] 69.2000 [69.1000, 69.7000]
ram32-upstream-1024 1.3636 [1.3613, 1.3648] 22.4615 [21.9063, 23.3003] 44.4000 [42.8000, 45.6000]
ram32-upstream-131072 203.4247 [203.3954, 203.4378] 23.5762 [23.3415, 24.5204] 42.3000 [40.7000, 42.7000]
ram32-upstream-32768 42.1757 [42.1616, 42.1790] 22.8831 [22.5282, 23.1798] 43.6000 [43.0000, 44.3000]
ram32-upstream-4096 4.7221 [4.7180, 4.7228] 21.7826 [21.7492, 23.5207] 45.8000 [42.4000, 45.9000]
ram32-upstream-8192 9.7800 [9.7797, 9.7814] 23.7654 [23.4857, 24.4217] 42.0000 [40.9000, 42.5000] 
block,sample,TTFT_s,TPOT_ms,prompt_tokens,reused,output_tokens,engine_prompt_ms,engine_decode_ms,engine_decode_tok_s,drafts_offered,drafts_accepted,finish
ram128-preview-1024,sample-1,1.526798245,14.091315321,1024,0,256,1498.3,3616.3,70.8,220,150,length
ram128-preview-1024,sample-2,1.525900942,14.155345925,1024,0,256,1496.6,3632.7,70.5,207,149,length
ram128-preview-1024,sample-3,1.527015871,13.497829846,1024,0,256,1497.7,3465.4,73.9,205,154,length
ram128-preview-131072,sample-1,184.274595868,14.811983443,131072,0,256,184174.0,3802.2,67.3,210,158,length
ram128-preview-131072,sample-2,184.319041520,14.324327907,131072,0,256,184172.0,3678.5,69.6,210,159,length
ram128-preview-131072,sample-3,184.293014582,15.475638576,131072,0,256,184194.3,3971.8,64.5,214,150,length
ram128-preview-32768,sample-1,37.734614095,14.340591427,32768,0,256,37675.9,3681.0,69.5,216,160,length
ram128-preview-32768,sample-2,37.740746810,13.579145609,32768,0,256,37673.3,3495.7,73.2,211,158,length
ram128-preview-32768,sample-3,37.718856161,14.163261479,32768,0,256,37659.8,3635.7,70.4,206,151,length
ram128-preview-4096,sample-1,4.499066725,14.570952547,4096,0,256,4466.0,3739.1,68.5,221,151,length
ram128-preview-4096,sample-2,4.502436898,13.443896934,4096,0,256,4469.5,3451.4,74.2,212,159,length
ram128-preview-4096,sample-3,4.499006665,13.055251984,4096,0,256,4465.0,3353.3,76.3,210,159,length
ram128-preview-8192,sample-1,8.962375381,14.307635360,8192,0,256,8924.8,3672.6,69.7,228,155,length
ram128-preview-8192,sample-2,8.962797106,14.423034254,8192,0,256,8926.8,3701.9,69.2,239,150,length
ram128-preview-8192,sample-3,8.961697640,14.411072915,8192,0,256,8923.7,3699.0,69.2,227,151,length
ram128-upstream-1024,sample-1,1.361792481,23.300253877,1024,0,256,1322.2,5976.0,42.8,220,150,length
ram128-upstream-1024,sample-2,1.362090691,22.456933773,1024,0,256,1321.9,5761.2,44.4,207,149,length
ram128-upstream-1024,sample-3,1.360803393,21.906111561,1024,0,256,1321.6,5620.6,45.5,205,154,length
ram128-upstream-131072,sample-1,203.392930999,23.569312193,131072,0,256,203242.7,6046.7,42.3,210,158,length
ram128-upstream-131072,sample-2,203.411487388,23.343555545,131072,0,256,203247.8,5989.7,42.7,210,159,length
ram128-upstream-131072,sample-3,203.424097860,24.512706317,131072,0,256,203259.1,6288.4,40.7,212,150,length
ram128-upstream-32768,sample-1,42.175370335,22.880901035,32768,0,256,42101.2,5871.0,43.6,216,160,length
ram128-upstream-32768,sample-2,42.162238366,22.521104293,32768,0,256,42088.5,5778.8,44.3,213,159,length
ram128-upstream-32768,sample-3,42.156530358,23.169628319,32768,0,256,42086.2,5944.4,43.1,206,151,length
ram128-upstream-4096,sample-1,4.720538725,23.526095427,4096,0,256,4676.0,6034.4,42.4,222,151,length
ram128-upstream-4096,sample-2,4.717103364,21.749313417,4096,0,256,4673.1,5581.0,45.9,212,159,length
ram128-upstream-4096,sample-3,4.719719660,21.781273404,4096,0,256,4675.2,5590.3,45.8,210,159,length
ram128-upstream-8192,sample-1,9.782985078,23.483234149,8192,0,256,9735.0,6024.3,42.5,228,155,length
ram128-upstream-8192,sample-2,9.789743559,24.415299041,8192,0,256,9741.5,6261.8,40.9,239,150,length
ram128-upstream-8192,sample-3,9.780472643,23.770459842,8192,0,256,9732.8,6097.5,42.0,227,151,length
ram32-preview-1024,sample-1,1.525920495,14.076147420,1024,0,256,1498.5,3612.1,70.9,220,150,length
ram32-preview-1024,sample-2,1.523796164,14.127060979,1024,0,256,1494.7,3625.9,70.6,207,149,length
ram32-preview-1024,sample-3,1.525502858,13.496443231,1024,0,256,1496.9,3464.1,73.9,205,154,length
ram32-preview-131072,sample-1,184.300920912,14.807121388,131072,0,256,184153.8,3801.7,67.3,210,158,length
ram32-preview-131072,sample-2,184.330562222,14.314907745,131072,0,256,184182.0,3676.7,69.6,210,159,length
ram32-preview-131072,sample-3,184.236893883,15.495199223,131072,0,256,184136.9,3976.7,64.4,214,150,length
ram32-preview-32768,sample-1,37.747930715,14.356828223,32768,0,256,37690.3,3685.5,69.5,216,160,length
ram32-preview-32768,sample-2,37.757844605,13.612727353,32768,0,256,37690.9,3504.1,73.1,211,158,length
ram32-preview-32768,sample-3,37.748622250,14.191877250,32768,0,256,37689.3,3643.6,70.3,206,151,length
ram32-preview-4096,sample-1,4.497383621,14.607577702,4096,0,256,4465.1,3749.4,68.3,221,151,length
ram32-preview-4096,sample-2,4.504398405,13.464493060,4096,0,256,4472.3,3457.4,74.0,212,159,length
ram32-preview-4096,sample-3,4.500201192,13.105146812,4096,0,256,4468.4,3365.9,76.1,210,159,length
ram32-preview-8192,sample-1,8.969425168,14.306654737,8192,0,256,8931.6,3672.8,69.7,228,155,length
ram32-preview-8192,sample-2,8.964368169,14.428386366,8192,0,256,8927.9,3703.2,69.1,239,150,length
ram32-preview-8192,sample-3,8.959855775,14.406340748,8192,0,256,8923.1,3697.4,69.2,227,151,length
ram32-upstream-1024,sample-1,1.364780472,23.300278260,1024,0,256,1323.1,5976.5,42.8,220,150,length
ram32-upstream-1024,sample-2,1.361335684,22.461492059,1024,0,256,1321.2,5762.1,44.4,207,149,length
ram32-upstream-1024,sample-3,1.363588390,21.906335526,1024,0,256,1323.0,5620.0,45.6,205,154,length
ram32-upstream-131072,sample-1,203.395417143,23.576186088,131072,0,256,203232.4,6048.3,42.3,210,158,length
ram32-upstream-131072,sample-2,203.437767783,23.341506740,131072,0,256,203272.8,5989.6,42.7,210,159,length
ram32-upstream-131072,sample-3,203.424709880,24.520412279,131072,0,256,203258.8,6289.4,40.7,212,150,length
ram32-upstream-32768,sample-1,42.175696562,22.883095364,32768,0,256,42102.0,5871.5,43.6,216,160,length
ram32-upstream-32768,sample-2,42.161633669,22.528247707,32768,0,256,42091.6,5780.3,44.3,213,159,length
ram32-upstream-32768,sample-3,42.178987773,23.179768632,32768,0,256,42108.1,5946.8,43.0,206,151,length
ram32-upstream-4096,sample-1,4.717975468,23.520662745,4096,0,256,4674.2,6033.4,42.4,222,151,length
ram32-upstream-4096,sample-2,4.722082125,21.749185486,4096,0,256,4676.9,5581.4,45.9,212,159,length
ram32-upstream-4096,sample-3,4.722833555,21.782557354,4096,0,256,4677.0,5590.1,45.8,210,159,length
ram32-upstream-8192,sample-1,9.781380199,23.485665894,8192,0,256,9733.0,6024.5,42.5,228,155,length
ram32-upstream-8192,sample-2,9.779996001,24.421670313,8192,0,256,9733.2,6262.8,40.9,239,150,length
ram32-upstream-8192,sample-3,9.779652208,23.765373193,8192,0,256,9731.3,6095.7,42.0,227,151,length

Sanitized startup excerpt, present in both B60 arms at both RAM limits:

strata mtp: draft head over 106299 tokens (137.9 MiB)
Current beta build, engine configuration and replay commands

Build/ctest executed. Launch/replay is reconstructed and source-checked only. Original timing/IPC helpers and frozen inputs await public attachment; independent reproduction is pending.

Use the pinned revisions and beta baseline frontend. Build:oneAPI2026.1; ggml headers:llama.cpp 3cf03257f219afbe7334045ff7c6a06ac68c627d:

cmake -S <SOURCE>/sycl -B <BUILD> -G "Unix Makefiles" \
  -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx \
  -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
  -DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_SYCL_PARITY=ON \
  -DSTRATA_GGML_DIR=<LLAMA_CPP>
cmake --build <BUILD> --target strata -j 12

For B70 preview, add -DSTRATA_SYCL_TLA_ATTN=ON -DSTRATA_SYCL_TLA_DIR=<SYCL_TLA> to configure. B60 uses AOT bmg-g21 with TLA off. Existing selected checks (B70 preview):

ctest --test-dir <PREVIEW_BUILD> --output-on-failure -R '^(esimd_moe_parity|gdn_parity|prefill_gdn_parity|gr_multi_parity|verify_parity|native_multi_parity|qsa_parity|quantize_act_parity|gemm_stride_parity)$'

B70 preview JSON; resolve paths. Expose one GPU (logical0); share <CALIBRATION_CACHE> across arms.

{"backend":"sycl","exe":"<BUILD>/strata","cwd":"<SOURCE>","args":["--pack","<PACK>","--native","<IQ2_XS_SHARD_1>","--ple-gguf","<IQ2_XS_SHARD_2>","--expert-profile","<SOURCE>/data/expert-profile.bin","--expert-cache","16000","--stream-experts","--vram-reserve-mib","1536","--no-prefill-borrow","--prefill","auto","--spec","4","--spec-min-p","0.5","--mtp","<MTP_RT>","--max-context","131584","--kv","int8","--adapt-every","8","--prompt-cache","0","--kv-resident","32768","--pcie-frac","0.30","--adapt-decay","1","--suffix-draft","3","--host-core","last"],"env":{"ONEAPI_DEVICE_SELECTOR":"level_zero:gpu","SYCL_CACHE_PERSISTENT":"0","ZES_ENABLE_SYSMAN":"1","STRATA_VERIFY_DEVICE_PLAN":"1","STRATA_VERIFY_NO_HOST":"1","STRATA_STAGER_THREADS":"12","STRATA_MIRROR_MIB":"16384","STRATA_DECODE_TIMING":"1","STRATA_SYCL_DEQUANT_GU_SG":"8","STRATA_SYCL_DEQUANT_GU_STATIC_TYPE":"1","STRATA_SYCL_DEQUANT_GU_STATIC_ROW":"1","STRATA_SYCL_PREFILL_COPY_PROFILING":"0","STRATA_SYCL_GR_LO_ONCE":"0","STRATA_SYCL_EXPERT_MULTI16":"0","STRATA_SYCL_IQ4_LOOKUP":"1","STRATA_SYCL_Q2_WORD_EXPAND":"1","STRATA_SYCL_MIRROR_ADAPT":"1","STRATA_SYCL_EXPERT_STAGE":"0","STRATA_SYCL_EXPERT_STAGE_GROUPS":"16","STRATA_SYCL_EXPERT_STAGE_COPY":"0","STRATA_SYCL_PACKED_SIGNS":"1","STRATA_SYCL_IQ3_XMX":"1","STRATA_SYCL_GDN_STATE_COMMIT":"1","STRATA_SYCL_ADAPT_WINDOW_PRESENCE":"1","STRATA_SYCL_IQ4_T4_XMX":"1","STRATA_SYCL_GR_NORM_BOUNDS":"1","STRATA_SYCL_IQ4_T4_ESIMD":"1","STRATA_SYCL_QSA_CHUNK_DOWN":"1","STRATA_SYCL_GR_CACHED_NORM":"3","STRATA_SYCL_DEQUANT_GU_BOUNDED":"1","STRATA_SYCL_IQ4_PAIR_REUSE":"1","STRATA_SYCL_GDN_QK_GRF256":"1","STRATA_SYCL_GDN_PREFILL_REG_QK":"1","STRATA_SYCL_F16_EXPERT_PAIR":"1","STRATA_SYCL_QSA_VALUE8_K8_VEC":"1","STRATA_SYCL_IQ4_Q8_SPLIT":"1","STRATA_SYCL_GU16_METADATA_QUARTET":"1","STRATA_SYCL_TLA_ATTN":"1","ZE_AFFINITY_MASK":"0","STRATA_PRIVATE_BMG_SINGLE_ACTOR":"1","STRATA_PRIVATE_GU_PREPARED":"1","STRATA_PRIVATE_GR_F32_REG":"1","STRATA_PRIVATE_GR_F32_PHASE":"down","STRATA_PRIVATE_GR_SILU_FROM_LO":"1","STRATA_PRIVATE_GR_LO_VEC4":"1","STRATA_PRIVATE_GR_2D_PAIR":"1","STRATA_PRIVATE_B60_NEXT_POOL":"1","STRATA_PRIVATE_MTP_Q8_SOA":"0","STRATA_PRIVATE_IQ4_HEAD_SOA":"0","STRATA_PRIVATE_B60_DECODE_LAYOUT":"1","STRATA_PRIVATE_PACKED_REGISTER":"1","STRATA_PRIVATE_GR_UP_ROWS16":"1","STRATA_PRIVATE_GR_Q8_UP32":"1","STRATA_PRIVATE_GU_XXS_PACKED":"1","STRATA_PRIVATE_DOWN_S4":"1","STRATA_PRIVATE_GU_GRID64":"1","STRATA_PRIVATE_IQ4_INT4_DPAS":"1","STRATA_PRIVATE_IQ3_S4":"1","STRATA_PRIVATE_IQ4_SSM_SOA":"1","STRATA_PRIVATE_GR_SOA_CACHE":"0","STRATA_PRIVATE_GU_GRID_S4":"1","STRATA_PRIVATE_GU_IQ1_PACKED":"1","STRATA_CACHE_DIR":"<CALIBRATION_CACHE>"},"tokenizer":"<PACK>/tokenizer","model_name":"qwen3.8-flash-next-iq2_xs","log":"<ENGINE_LOG>","host":"127.0.0.1","port":8095,"gpu":0,"open_browser":false,"parallel":1,"sampling":{"temperature":0.0,"seed":42,"top_p":1.0,"top_k":64,"min_p":0.0,"repetition_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"penalty_last_n":64}}

B70 upstream: expert-cache auto; remove all STRATA_SYCL_* / STRATA_PRIVATE_* env keys, retaining the other ten. B60: both arms use expert-cache auto, STRATA_MIRROR_MIB=32768, gpu_order="as_given"; preview STRATA_SYCL_TLA_ATTN=0. Other fields match. Draft vocab106299. Container:16CPU/cpuset0–15; RAM tiers as tabulated, swap unrestricted, shared page cache retained.

Launch:

python <BASELINE_SOURCE>/serve/server.py --engine strata --config <RESOLVED_CONFIG_JSON> --host 127.0.0.1 --port 8095

Restart per arm/RAM/length block; clear inherited STRATA_* before launch. After readiness, replay from the frozen payload directory:

N=32768
mkdir <NEW_OUTPUT_DIR>
i=0
for f in short-warmup code-${N}-warmup code-${N}-warmup code-${N}-warmup code-${N}-1 code-${N}-2 code-${N}-3
do
  i=$((i+1))
  curl --fail-with-body --no-buffer http://127.0.0.1:8095/v1/chat/completions \
    -H 'Content-Type: application/json' --data-binary "@${N}/${f}.json" > "<NEW_OUTPUT_DIR>/${i}.sse"
done

Use N=1024/4096/8192/32768/131072. Exclude1–4, measure5–7; verify input/output counts, reuse0, length finish, text/token IDs. Timing definitions are above; curl alone does not measure them. Actual instrumented launcher: python -I -B run_block.py --root <BLOCK_ROOT> --arm <N>-single --readiness-timeout 1200 --request-timeout 1200. Harness/corpus publication pending.

Area

[Engine] (SYCL speed and kernels; E2E validation)

Desktop app, Terminal or source?

Source builds inside Docker.

What happened

This is the existing combined single-GPU performance demonstration, being revalidated after the 0.2.0-beta tracker reset. The historical 0.1.45 results below remain historical. The preview is a runnable integration for E2E discussion; focused optimization PRs would follow if upstream is interested.

Reproduction status

Tested locally using unmodified 0.2.0-beta / engine 0.1.45.3 source. The initial functional check completed one short warmup (19 input tokens, 2 output tokens), then six fresh 1,024-token requests (three full-length warmups and three recorded requests, each generating 256 tokens), with no prompt-prefix reuse and a clean engine unload. It overlapped candidate compilation; all timings from that check are excluded from performance claims.

The original upstream Battlemage default remains enabled: STRATA_PREFILL_MMQ=auto selects ESIMD INT8 prompt-expert products using the measured crossover, with FP16 fallback. The upstream log confirms 498 INT8 groups and 2 FP16 groups at the logged checkpoint. This upstream feature is not claimed as a preview optimization. Its quantized activations can change outputs relative to the older FP16 path; the old results below are not a numerical reference for this new version.

The pure upstream comparator uses the listed custom serving configuration and its original dispatch. This is not an untouched installer-default benchmark. It uses --expert-cache auto, while the preview uses --expert-cache 16000; with STRATA_VERIFY_NO_HOST=1, upstream disables adaptive swaps, whereas the preview explicitly enables its mirror-exchange path. Both keep fixed --pcie-frac 0.30.

The preview has passed its two SYCL engine builds and nine selected regression cases; the completed comparisons are shown above. The requested B70 32/64/128 GiB and B60 32/128 GiB five-length matrices are complete; other validation gaps remain stated below.

Strata version

Measured beta source: Strata 0.2.0-beta / engine 0.1.45.3, upstream commit 3d1fb48090e8f2cefc22d7b5845fb7f5b71f33fa. Latest observed upstream main is e88ee39d1d6be6e3364183af3b39b62a7ed6f211; its sycl/, src/, include/, serve/, data/ and root CMakeLists.txt trees are identical to the beta source. This is verified source equality, not a separate latest-main binary test.

Current draft PR #3153 head is cb436853ab2ee238fa579a3a4f42a0412b46c439, rebased onto main at f57c0adcb31c00f3f216550c004dac4e713a9f70. The builds and nine tests ran at c3ed1d047768d94f43d7384ccf740a214f5a53dd; the current head has identical engine and server source trees. The original performance tables were measured on engine 0.1.45 at 477ac13cbe8e582c20484871ce459d20d82df09d, with preview 3440db820d77cc7e71b4a30cc82e966290de1f2d.

Graphics card, memory and system

One Intel Arc Pro B70, 32 GB VRAM, device 0xE223; Intel Xeon w5-3435X (32 logical CPUs), 16 CPU quota / 15 workers; Ubuntu 24.04.2 LTS, kernel 6.17.0-1007-intel. Host reports 503.36 GiB usable RAM. The 32/64/128 GiB values in the tables are container memory limits, not installed-RAM capacity claims. oneAPI 2026.1; source-built SYCL AOT bmg-g31; image omix 0.4.0.

Model used

This measurement uses the original full-expert IQ2_XS release, rather than the pruned Coder model or the Swift/Unsloth alternatives.

Optimizations in the preview and existing upstream PRs

The first five rows map to our existing focused upstream submissions. The remaining rows describe additional work present in the preview; they do not claim that an earlier PR already contains those extensions. All five linked focused upstream PRs were confirmed open on 2026-10-20; status does not establish that their changes were merged. The E2E table measures the combined configuration; individual PR percentages are not added together.

Optimization Scope in this preview Our existing upstream PR
IQ4_XS MMVQ pair reuse, Q8 split and packed bit-select lookup Reuse weight/header loads and reduce IQ4 decoding work on eligible shapes #2555
INT8 QSA decode chunk-down and vector key/value loads Shorter decode attention/reduction paths on eligible layouts #2582
Synchronous adaptive expert-mirror replacement Exchange resident/mirrored experts while preserving logical expert identity #2583
Paired FP16 expert prefill, bounded IQ2 gate/up decoding and metadata reuse Reduce prompt-path launches and decoding work #2584
Q2_0 expert-down packed-word expansion Replace per-byte/codebook expansion on the eligible down path #2746
Packed resident expert layouts with raw host-mirror storage Integrate the packed decode path with single-device streamed experts Not separately submitted from this preview
Coalesced host-word exchange for packed resident experts Extend mirror replacement to the packed-device/raw-host layout Additional work beyond #2583; no separate upstream PR yet
Prepared and register-based IQ gate/up/down variants Prepare/reuse packed inputs and metadata; include format/shape-specific variants Not separately submitted from this preview
IQ3 packed/S4 and IQ4 INT4-DPAS paths Additional eligible decode projection variants Not separately submitted from this preview
IQ4 SSM weight-layout cache Retain the selected SSM projection cache Not separately submitted from this preview
GR cached normalization and bounded/vector/register projection variants Reuse normalized inputs and low-rank results; reduce eligible projection work Not separately submitted upstream from this preview
GR up rows16/Q8-up32 and paired 2D variants Additional GR projection implementations for eligible cases Not separately submitted from this preview
GDN state-only commit and QK register variants Avoid unnecessary commit work and use eligible QK paths Not separately submitted from this preview
Reusable decode scheduling pool and MTP/verify integration Integrate the selected expert/projection paths with speculative verification Not separately submitted from this preview
Exact FP32 TLA prompt attention Optional B70/g31 prompt-attention path Not separately submitted from this preview
Optional GR SOA, MTP Q8 SOA and IQ4 head SOA caches Available in the code but disabled in the measured B70 configuration Not separately submitted from this preview

A control being enabled does not mean every shape-specific branch executes. These rows are an implementation inventory, not independent claims that every variant contributes to the measured gain.

Contribution and measurement status

This combined engine preview remains a draft for scope and E2E discussion; maintainer agreement to merge it has not been obtained. Some integrated controls still default on, and an all-controls-off byte-identical default path has not been established. The current evidence does not demonstrate full compliance for merge review. Focused optimization submissions and a separate results-only report would need the applicable checks and sanitized raw evidence.

These are custom community measurements with thinking off, greedy sampling and a 256-token cap. They do not implement the draft standard benchmark protocol. Client TPOT-based tok/s is distinct from engine decode throughput. The published historical mean tables are not a substitute for the per-request evidence and median/range summaries requested by the current community guide.

Historical engine 0.1.45 tables, configuration and related-work notes (superseded)

Historical engine 0.1.45 results: single-GPU IQ2_XS E2E demonstration

Historical measured preview: 3440db820d77cc7e71b4a30cc82e966290de1f2d. Upstream draft PR #3153 now contains the beta rebase identified above; the old tables are retained with their original revisions.

This issue and the linked draft PR demonstrate end-to-end performance of the combined optimizations on the measured single-GPU workloads. The preview branch is a runnable integration snapshot for reproducing and discussing the result. It is not proposed as one patch to merge in full.

If upstream is interested in these results, the work will be split into individual optimization PRs, following the same approach as the focused submissions listed below. Each such PR can then be reviewed and validated independently, including its dispatch, fallback, correctness and performance scope.

One Intel Arc Pro B70 (32 GB VRAM) running the original Qwen3.8-Flash-Next IQ2_XS measures 94.12–106.12 client tok/s across 1K–128K prompts with a 32 GiB container RAM limit. The 32K result is 98.06 tok/s; the 128K result is 94.12 tok/s. “~100 tok/s” describes this range. The configured container limit is not proof that a machine with 32 GB installed RAM can run this workload: shared model page cache was retained and the full physical-memory accounting has not been established.

Historical E2E performance results (engine 0.1.45)

B70, 32 GiB container RAM limit

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Preview tok/s
1K 1.167 1.169 -0.18% 13.711 9.647 29.64% 103.66
4K 3.493 3.039 12.99% 13.468 9.423 30.04% 106.12
8K 7.093 5.952 16.08% 13.829 9.917 28.29% 100.84
32K 29.982 24.577 18.03% 14.200 10.198 28.18% 98.06
128K 140.328 117.213 16.47% 14.928 10.625 28.83% 94.12

B70, 64 GiB container RAM limit

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Preview tok/s
1K 1.168 1.169 -0.11% 13.714 9.655 29.60% 103.57
4K 3.496 3.038 13.10% 13.480 9.420 30.12% 106.15
8K 7.090 5.945 16.14% 13.818 9.912 28.27% 100.88
32K 30.007 24.559 18.16% 14.202 10.192 28.24% 98.12
128K 140.399 117.113 16.59% 14.932 10.626 28.84% 94.11

B70, 128 GiB container RAM limit

Prompt tokens upstream TTFT s Preview TTFT s TTFT reduction upstream TPOT ms Preview TPOT ms TPOT reduction Preview tok/s
1K 1.171 1.170 0.08% 13.723 9.665 29.57% 103.46
4K 3.482 3.038 12.73% 13.493 9.424 30.16% 106.12
8K 7.072 5.948 15.89% 13.846 9.920 28.35% 100.80
32K 29.940 24.598 17.84% 14.219 10.191 28.33% 98.13
128K 140.048 117.176 16.33% 14.939 10.625 28.88% 94.11

B60, 128 GiB container RAM limit

Prompt Upstream TTFT s Preview TTFT s TTFT reduction Upstream TPOT ms Preview TPOT ms TPOT reduction Preview tok/s
1K 1.605 1.664 -3.67% 22.327 14.220 36.31% 70.32
4K 5.067 4.669 7.87% 22.891 14.006 38.82% 71.40
8K 10.454 9.289 11.15% 23.752 14.392 39.41% 69.48
32K 44.911 39.016 13.13% 23.311 14.442 38.04% 69.24
128K 214.801 189.609 11.73% 23.721 15.427 34.97% 64.82

Measurement method and scope

Measured preview commit 3440db820d77cc7e71b4a30cc82e966290de1f2d against pure upstream 477ac13cbe8e582c20484871ce459d20d82df09d, engine 0.1.45.

Three measured requests per configuration/length, after one short and three full-length warmups; 256 generated tokens, thinking off, seed42 and no prompt-prefix reuse. Negative reduction means a latency regression.

K=1024. Each full request generates 256 tokens. Each configuration/length has three measured requests after one short and three full-length warmups. Thinking is off, seed42, no prompt-prefix reuse. TTFT is request submission to the first nonempty answer-text SSE delta; TPOT is the first-to-last nonempty delta interval divided by 255. Client tok/s is 1000 / mean(TPOT_ms). Reduction percentages use unrounded means: (upstream - preview) / upstream * 100; positive means lower latency.

The pure upstream comparator is engine 0.1.45 at 477ac13cbe8e582c20484871ce459d20d82df09d, and the preview is based on the same commit. Both use identical inputs, sampler and fixed MTP parameters. Actual speculative work is allowed to vary and is not presented as equal-work kernel timing. All 35 paired requests in the completed 128 GiB series, including warmups, match text and token IDs. These are warmed E2E characterization results; model load and profiled requests are excluded. Shared model page cache was not flushed, so a container RAM limit is not a minimum installed-RAM claim.

105 paired requests, including warmups, have matching text and token IDs in the completed comparisons. Actual MTP work is not normalized. The tables show whole-engine behavior, including adaptation and actual cache capacity.

B70 retains the selected fixed expert-cache request16000 and mirror budget16384 MiB. B60 uses a native bmg-g21 build, TLA OFF, auto expert-cache sizing and mirror budget32768 MiB on both executables. These platform differences are part of the reported configurations, not isolated causal GPU comparisons.

RAM values are container limits in GiB (N×2³⁰ bytes). Shared model page cache was retained. These measurements do not establish minimum installed system RAM.

Maximum observed swap sample across completed cells: 0 bytes. New cgroup OOM / OOM-kill events across completed cells: 0 / 0.

Related work and submission scope

This is a combined E2E demonstration for one Arc Pro B70, not a proposal to merge the whole preview as one change. If upstream is interested, we would split the remaining changes into focused PRs, following the earlier submissions, and coordinate with existing work first. The combined gain is not the sum of individual PR gains.

A 2026-10-19 review of upstream found related work below. We did not find another report with this same single-B70 IQ2_XS preview and the 1K–128K upstream/optimized TTFT/TPOT comparison. Existing single-B70 reports #2922 and #2867 use IQ3_S and IQ3_XXS respectively; #1054 includes older single-card IQ2_XS results within a two-card bug report. Their hardware, workloads and measurement boundaries differ, so their TPS are not direct comparators.

Area in this preview Our focused submission Related upstream work and overlap
IQ4_XS MMVQ #2555 #2804 / issue #2803 use KREG kernels at the same MMVQ entry points; different activation reuse/reduction. #2437 changes an expert-dot lookup left unchanged by this preview. Coordinate these distinct paths.
INT8 QSA chunk/vector decode #2582 #2806 / issue #2805 partially overlap in attention; also add slot batching, scores/top-k and active-chunk merge. Reconcile shared dispatch hooks.
Synchronous mirror adaptation and packed resident/raw host integration #2583 Packed/raw storage and coalesced exchange extend our existing PR. These preview-specific extensions have not been independently submitted yet.
Paired FP16 experts and bounded IQ2 prompt decoding #2584 #1215 explored a different padded 16-expert oneMKL batch and was closed pending upstream E2E verification.
Q2_0 packed-word expansion and further IQ decode variants #2746 #2802 / issue #2801 have directly overlapping Q2 shift/mask expansion arithmetic, plus broader IQ/lane kernels. Reconcile that common subset in the existing PRs; packed/register variants are separate implementation choices.
GR normalization/projection reuse and register variants No separate upstream submission from this preview yet #2812 / issue #2811 fuse a BF16 read; #2810 implements Q8_0-weight reads; #2488 targets A770 GR fusion. Same bottleneck, different paths. This preview's Q8-up32 retains BF16 weights and produces Q8_1 activations.
GDN state-only commit and QK registers No separate submission from this preview yet #2755 / #2586 optimize recurrence and batch commits across layers. Coordinate common state replay; the preview currently selects a per-layer state-only path.
FP32 prompt attention No separate submission from this preview yet #2808 / issue #2807 use 8-query unions and FP16 joint_matrix products. This preview uses FP32 with separate device/shape guards; numerical contracts differ.

Our five linked focused upstream PRs were confirmed open on 2026-10-20. “No separate submission” refers only to this preview's remaining delta, not to an absence of related work. The scheduling proposal in issue #2416 is another complementary direction, not part of this preview.

Existing upstream foundations are not claimed as new preview work: Q6_K SWAR/planar/aligned loads from #2340 are already in the historical baseline. PR #2340 is now closed; its state does not change that source ancestry. IQ3_S wide #1774, grouped XMX #1711, and GR foundations #2193 / #2203 are also already present. The optional GR SOA, MTP Q8 SOA and IQ4 head SOA caches are disabled in the reported B70 configuration.

Public source and configuration

  • Upstream draft PR #3153: complete integration snapshot for E2E discussion.
  • Branch: preview/sycl-bmg-iq2xs-100tps-20261019.
  • Source: 3440db820d77cc7e71b4a30cc82e966290de1f2d, based on 477ac13cbe8e582c20484871ce459d20d82df09d.
  • B70 build: oneAPI 2026.1, AOT bmg-g31, normal tracing disabled, optional TLA prompt attention enabled.
  • Max context 131584; INT8 KV resident 32768; MTP spec4/min-p0.5/suffix3; prompt cache 0; neutral sampler / seed42.
  • Selected expert-cache request 16000, actual 16780 resident / 7796 mirror; host-mirror budget 16384 MiB; VRAM reserve 1536 MiB;16 CPU quota / 15 workers.
  • GR SOA, MTP Q8 SOA and IQ4 head SOA caches are disabled; the selected pool/IQ3/SSM paths remain enabled. Full runtime configuration and build command are in the draft PR.

Activity

  1. xiangyuT
    changed the title [-][SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: single-B70 E2E preview (~100 tok/s)[/-] [+][SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/+] on Oct 19, 2026

  2. Niko1221 commented on Oct 19, 2026

    @Niko1221
    Niko1221
    on Oct 19, 2026
    Owner

    Thanks for taking the time to report this.

    We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

    To reopen it takes about two minutes:

    1. Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
    2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
    3. Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
    4. Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

    The rules are in CONTRIBUTING.md.

  3. Niko1221
    closed this as not plannedon Oct 19, 2026
  4. xiangyuT
    changed the title [-][SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/-] [+][Benchmark] [SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/+] on Oct 19, 2026

  5. xiangyuT commented on Oct 19, 2026

    @xiangyuT
    xiangyuT
    on Oct 19, 2026
    Author

    /reopen

  6. github-actions
    reopened this on Oct 19, 2026

  7. github-actions commented on Oct 19, 2026

    @github-actions
    github-actionsBot
    on Oct 19, 2026 – with GitHub Actions

    Reopened, thank you. A maintainer will pick it up from here.

  8. xiangyuT
    changed the title [-][Benchmark] [SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/-] [+][Engine] [SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/+] on Oct 19, 2026
  9. xiangyuT
    changed the title [-][Engine] [SYCL][Perf] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/-] [+][Engine] [SYCL] Qwen3.8-Flash-Next IQ2_XS: ~100 tok/s on B70 (32 GB VRAM, 32 GiB container RAM limit)[/+] on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions