From 13bd5fec011ff725b071cee6808c2c4a704e2ac9 Mon Sep 17 00:00:00 2001 From: LocalXPU <337401568+LocalXPU@users.noreply.github.com> Date: Sun, 4 Oct 2026 19:30:38 -0500 Subject: [PATCH 1/3] docs, setup: Arc Pro B60 rows, B60 notes, PCI id e211 INTEL_ARC.md: "What has been run" rows for 2x and 1x Arc Pro B60, plus notes for the card: PCI id 8086:e211 (bmg-g21), setvars.sh stopping at OCL_ICD_FILENAMES under `set -u`, running natively without Docker, and the layer-split startup warning that counts the other card's layers. sycl/setup_intel.py: the B60 table entry was e221, which the kernel's xe id list files under the BMG-G31 cards (0xE220-0xE223). Both Arc Pro B60 cards measured here are 8086:e211, so setup named them "Intel GPU e211 (xe)" with a guessed VRAM size. e211 is added; e221 stays, since there is no card here to say it is wrong. tools/test_setup_sycl.py checks the table. COMMUNITY_BENCHMARKS.md and bench/results/2026-10-04-community-2x-arc-pro-b60/: the report, in the template's format, with the server configs, engine logs and per-request timings. --- .../README.md | 70 +++++++++++++++++++ .../data/B2.run.txt | 54 ++++++++++++++ .../data/bench.py | 41 +++++++++++ .../data/c2-fib1.json | 1 + .../data/c2-long.txt | 8 +++ .../data/c2-long2.txt | 26 +++++++ .../data/c2-short.txt | 35 ++++++++++ .../data/coder2.engine.txt | 66 +++++++++++++++++ .../data/coder2.json | 47 +++++++++++++ .../data/i2-long.txt | 8 +++ .../data/i2-long2.txt | 23 ++++++ .../data/i2-short.txt | 27 +++++++ .../data/iq2b.engine.txt | 69 ++++++++++++++++++ .../data/iq2b.json | 47 +++++++++++++ docs/COMMUNITY_BENCHMARKS.md | 3 + docs/INTEL_ARC.md | 16 +++++ sycl/setup_intel.py | 3 +- tools/test_setup_sycl.py | 21 ++++++ 18 files changed, 564 insertions(+), 1 deletion(-) create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/README.md create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/B2.run.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/bench.py create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-fib1.json create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long2.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-short.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.engine.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.json create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long2.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-short.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.engine.txt create mode 100644 bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.json diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md b/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md new file mode 100644 index 00000000000..1c57d1cd00f --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md @@ -0,0 +1,70 @@ +# Community benchmark on 2x Intel Arc Pro B60 + +Measured on 2026-10-04 by LocalXPU. This is the SYCL port (`sycl/`) on two Arc Pro B60 cards, one request at a time, at an 8,192-token context. It is a short test: two prompt sizes, 2 to 3 requests each. It is not a 4K/32K/128K sweep. + +## Hardware and software + +- 2x Intel Arc Pro B60 24 GB (`8086:e211`, 456 GB/s each), PCIe 3.0 x8 per card; Ryzen 5 5600; 62 GB DDR4; storage type not recorded +- Ubuntu 24.04, kernel 6.17 (`xe` driver), Level Zero V2, compute runtime NEO 26.09.37435.12, oneAPI DPC++ 2026.1.1, oneMKL 2026.1.0 +- Strata `6f32ec0` plus two `sycl/` fixes (the build fix for #559/#626 and the `cudaStreamQuery` fix in the ring wait); engine reports `0.1.39-sycl`; AOT `-DSTRATA_SYCL_AOT=bmg-g21`; source build, no Docker +- Nothing else ran on the GPUs. The kernel log was watched for `xe` resets during every run: none. + +## Model and configuration + +- Coder IQ1_M (`Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M`, shard hashes checked against the Hugging Face LFS ids), `--layer-split 24` +- Flash-Next IQ2_XS (`Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS`), `--layer-split 25` +- Packs from `tools/iq_pack.py`, MTP layer from `tools/mtp_fetch.py` / `mtp_pack.py` / `mtp_rt.py`, profiles from `data/` +- Context 8,192; KV int8; `--spec 4 --spec-min-p 0.5`; greedy (`temperature 0`), `enable_thinking: false`; no vision + +`data/coder2.json` and `data/iq2b.json` are the exact server configs (engine arguments and environment); the engine's startup and stage lines are in `data/coder2.engine.txt`, `data/iq2b.engine.txt` and, for the one-card run, `data/B2.run.txt`. Served with +`sycl/serve/server_intel.py --engine strata --config --port 8085`. + +## Method + +Requests went through `/v1/chat/completions`, one at a time. Prompt and decode speed are the server's own `timings` object +(`prompt_per_second`, `predicted_per_second`), per request; `bench.py` prints them (copy in `data/`, raw output in `data/c2-*.txt`, `data/i2-*.txt`). +Model loading is not included. Three prompts: + +- short: "Write a Python function that returns the n-th Fibonacci number" (23 prompt tokens cold, 7 fresh after the cached prefix) +- 2K code: about 2,000 tokens of the repository's `tools/*.py` plus "write a binary search tree class" (1,922 prompt tokens, none reused); the second turn reuses the first +- 2K prose: about 2,100 tokens of source plus "say what this code is for" (2,129 prompt tokens, none reused); the repeat reuses 2,122 of them + +## Results + +All experts were resident in VRAM on both cards in both runs (engine line `layer split: 100% of the experts resident`), so `STRATA_VERIFY_NO_HOST=1` was set. + +| Configuration | Prompt tokens (fresh) | Reused | Generated | Requests | Prompt tok/s | Decode tok/s | Drafts accepted | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| Coder IQ1_M, short | 23 / 7 | 0 / 16 | 159 | 3 | 50.4 cold, 74-80 | 52.8 first, 56.6 and 56.6 | 112/138, 114/133 | +| Coder IQ1_M, 2K code | 1,922 | 0 | 256 | 1 | 425.3 | 57.2 | 186/208 | +| Coder IQ1_M, 2K code, next turn | 21 | 2,178 | 256 | 1 | 61.4 | 71.4 | 209/212 | +| Coder IQ1_M, 2K prose | 2,129 | 0 | 128 | 1 | 427.9 | 40.1 | 70/98 | +| Coder IQ1_M, 2K prose, repeat | 7 | 2,122 | 128 | 1 | 78.1 | 41.3 | 70/98 | +| IQ2_XS, short | 23 / 7 | 0 / 16 | 112 | 2 | 50.4 cold, 81.7 | 58.6 and 61.3 | 84/88, 84/89 | +| IQ2_XS, 2K code | 1,922 | 0 | 256 | 1 | 445.1 | 55.5 | 183/213 | +| IQ2_XS, 2K code, next turn | 284 | 1,915 | 256 | 1 | 177.0 | 72.5 | 211/217 | +| IQ2_XS, 2K prose | 2,129 | 0 | 104 | 1 | 436.4 | 38.8 | 56/87 | +| IQ2_XS, 2K prose, repeat | 7 | 2,122 | 104 | 1 | 79.3 | 40.0 | 56/87 | + +The two prose rows are slower because fewer drafts were accepted (71% and 64%), not because of the context: the 256-token code reply at the same context ran 57.2 and 55.5. +Time to first token was not measured separately. + +One card, Coder IQ1_M, 4,042 of 12,288 experts mirrored in pinned RAM (`STRATA_MIRROR_MIB=8192`, `ONEAPI_DEVICE_SELECTOR=level_zero:0`), +by-hand `--tokens` run, 11-token prompt, 128 greedy tokens: **11.9 tok/s** (11.2 to 11.9 over five builds that differ only in experiments), coherent Python. +Without the ring-wait fix this configuration stopped at layer 15 with `never rang (graph finished)` in 5 of 5 tries. +The engine's stage table shows `wait for rings` at 186.8 ms per round, about 3.9 ms per layer. It drops to 72.9 ms and 25.1 tok/s with `kSpinMax` 2,000 in +`sycl/include/strata/sycl_doorbell.hpp`, which suggests the GPU does not see the host's flag stores during the spin kernel and waits out the bound. That change +risks proceeding before a slow host answers, so it is not part of this report. + +Memory after load (`xpu-smi`, MiB, card 0 / card 1): Coder 14,592 / 17,456; IQ2_XS 20,246 / 20,573. The tightest card kept 3.7 GB free of the 24 GB with `--vram-reserve-mib 1536`. + +## Correctness and limitations + +- Both models, `/v1/chat/completions`: a Fibonacci function executed against the first ten values and `fn(50) == 12586269025`; a Euclid `gcd` executed on four pairs; a palindrome check (IQ2_XS); `sum(i*i for i in range(1,11))` answered `385`. All passed. +- A three-turn conversation (a name and a number, then arithmetic on it, then a lambda using both) was answered correctly by both models, and the conversation cache reused 41 and 74 tokens. +- No NaN, garbage or repetition loop in any output. +- Not done: the 19-token chat-templated prompt from the docs (no token-for-token comparison with the reference text), streaming, the Anthropic route, tool calls, contexts above 2.2K, `--layer-split auto`, a run with `STRATA_VERIFY_NO_HOST=1` and experts outside VRAM. +- Kernel tests (`ctest`, one B60): 21 of 25 pass; `iq_parity` passes for all 10 formats once its fixtures exist; `ple_parity` needs the Q2_0 shard; `s2_expert_grouped_parity` fails 4 bitwise comparisons while its host-reference check is within tolerance; `conversation_snapshot_test` aborts with `munmap_chunk(): invalid pointer`. +- One `xe` GT reset on card 0, during the first `iq_parity` run (it did not finish in 300 s and was killed by a timeout; the same binary then passed in under 90 s). The card recovered without a reboot. + +Paths in `data/` were shortened: `` is the directory with the models, packs and MTP layer, `` the Strata source tree. diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/B2.run.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/B2.run.txt new file mode 100644 index 00000000000..aa646dd248d --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/B2.run.txt @@ -0,0 +1,54 @@ +IDS=7734,264,12654,709,421,4523,279,307,7324,76938,1324 +strata generate: PCIe probe: 7.0 GB/s host->device (best of 7.0 7.0 7.0 7.0) -> pcie_frac 0.19 (default 0.55) +strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows) +strata generate: native pack: /packs/coder-iq1_m experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB) +strata generate: 1406 MiB of weights loaded from /packs/coder-iq1_m (302 canonical tensors skipped: served natively) +strata generate: 300 native projection matrices, 2018.88 MiB of weights +strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off) +strata generate: profile data/expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots +strata generate: PLE on, table 320001536 rows of /models/coder-iq1_m/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf +strata mtp: draft layer loaded, 802 MiB of VRAM (experts 675, dense 111), files read in 0.45 s (1746 MiB/s) +strata generate: experimental native Q5_K head, 521472000 bytes +strata generate: GPU 0: Intel(R) Arc(TM) Pro B60 Graphics, compute capability 20.1 +strata generate: WARNING: this engine was compiled with CUDA 12.8 headers but loaded a CUDA 0.2 runtime (libcudart): GPU properties can read wrong and some kernels go unused. Rebuild it against one toolkit (cmake -DCUDAToolkit_ROOT=, with its libcudart.so present) (#542) +strata generate: experts streamed from the GGUF on demand (--stream-experts): no host arena; every expert must fit the VRAM cache +strata generate: expert cache auto: 18.90 GiB free, 1536 MiB reserved (+218 MiB for the draft head) -> 6331 slots +strata generate: expert cache 8246 slots, 15.70 GiB of VRAM; policy is +strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU, + so a reply can differ slightly from a run without the cache (same quality: + bench/results/2026-09-27-cache-parity). + PROFILE, ranked by routing frequency, no eviction. +strata generate: profile fill: 15.70 GiB in 2.5 s (6.79 GB/s, pipelined) +strata generate: pre-filled 8246 of 8246 slots from the profile; slot 0 verified +strata generate: 4042 of 4042 experts missing from VRAM mirrored in pinned host memory (7.72 GiB, 1.3 s); the GPU reads them over PCIe +strata generate: R4 hit path ON - resident experts are computed on the GPU +strata generate: 5 expert-pool workers + the host thread +strata generate: session is up (engine 0.1.39-sycl) +strata generate: sampling greedy +strata generate: token graph hit path: 8246 resident experts, decided on the device +strata generate: prompt path allocates its own buffers (no cache slots to borrow) +strata mtp: draft head over 106299 tokens (212.9 MiB) +strata generate: prefill 10 tokens in 1 chunks, 563.9 ms (17.7 tok/s); experts streamed 354 (354 by DMA, host 5.2 ms), resident 1689; PLE 2.7 ms +strata verify: window up to 6 tokens, 74.0 MiB of device buffers +strata verify: captured the 1-token window (upload , sync ) +strata verify: captured the 2-token window (upload , sync ) +strata verify: captured the 3-token window (upload , sync ) +strata verify: captured the 4-token window (upload , sync ) +strata verify: captured the 5-token window (upload , sync ) +strata verify: captured the 6-token window (upload , sync ) +strata generate: window and draft graphs captured in 125 ms +speculation 50 rounds of 6, drafts accepted 81 of 98 (0.827), 2.62 tokens per round +window sizes T1:11 T2:8 T3:3 T4:28 T5:0 T6:0 (min draft probability 0.50) +suffix drafts 3 windows, drafts accepted 8 of 9 +accepted per round 0:14 1:10 2:7 3:19 4:0 5:0 +verify window wait for rings 186.845 pool 10.966 host 1.142 commit 9.963 ms/round; CPU experts 2.50 distinct / 3.33 routed per layer +pool multi gate/up 6.183 quantize 0.025 down 2.868 ms/round; 27.1 GB/s over the rows phases; CPU pool call 10.958 ms/round +dispatch plan 0.100 activation quantize 0.254 jobs 1.514 run 9.080 ms/round +adaptive tier 1069 experts swapped into the VRAM tier (every 4 rounds, 0.839 ms/round) +pcie experts 0.21 distinct experts per layer read over PCIe (share 49/256 of the misses) +mtp 10.095 ms/round drafting (49 rounds), MTP prompt 6.8 ms, 1015 MiB of VRAM +prompt : 7734 264 12654 709 421 4523 279 307 7324 76938 1324 +output : 1608 48575 13 271 727 15336 1393 1590 198 262 413 307 2564 220 16 25 198 285 460 307 198 262 460 15336 1393 12 16 8 478 15336 1393 12 17 8 271 760 709 3294 369 264 4145 29812 7873 314 279 76938 8240 13 4213 11 424 369 524 10727 364 3349 2663 314 307 1521 424 95272 3205 942 279 1788 2663 5081 2942 13 1946 7042 14588 11 567 628 958 20781 1954 310 3436 8334 16100 2663 13 248046 198 248045 846 198 7734 264 12654 709 421 4523 279 307 7324 76938 1324 1608 48575 248046 198 248045 74455 198 248068 198 760 1156 6587 264 12654 709 421 4523 279 307 7324 76938 1324 1608 48575 13 1061 369 +decode 128 tokens in 10735.2 ms -> 11.92 tok/s +prefill 10 tokens in 563.9 ms -> 17.73 tok/s (time to first token 809.0 ms) +EXIT=0 diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/bench.py b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/bench.py new file mode 100644 index 00000000000..41b46ad7d5f --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/bench.py @@ -0,0 +1,41 @@ +import json,sys,re,urllib.request,time,glob +URL="http://127.0.0.1:8085/v1/chat/completions" +def ask(msgs,max_tokens=256): + body={"messages":msgs,"max_tokens":max_tokens,"temperature":0,"chat_template_kwargs":{"enable_thinking":False}} + t=time.time() + r=urllib.request.urlopen(urllib.request.Request(URL,json.dumps(body).encode(),{"content-type":"application/json"}),timeout=900) + d=json.load(r); d["wall"]=time.time()-t + return d +def show(tag,d): + tm=d["timings"];u=d["usage"] + print(f"[{tag}] prompt_n={tm['prompt_n']} cache_n={tm['cache_n']} prompt {tm['prompt_per_second']:.1f} tok/s | decode {tm['predicted_n']} tok {tm['predicted_per_second']:.1f} tok/s | drafts {tm.get('draft_n_accepted')}/{tm.get('draft_n')} | wall {d['wall']:.1f}s | finish {d['choices'][0]['finish_reason']}") +def code(text): + m=re.search(r"```python\n(.*?)```",text,re.S); return m.group(1) if m else text +def run_tests(src,tests): + g={} + try: + exec(src,g); exec(tests,g); return "PASS" + except Exception as e: return f"FAIL {e!r}" +mode=sys.argv[1] +if mode=="short": + for i in range(2): + d=ask([{"role":"user","content":"Write a Python function that returns the n-th Fibonacci number"}],256); show(f"fib run{i}",d) + print(d["choices"][0]["message"]["content"]) + print("fib tests:",run_tests(code(d["choices"][0]["message"]["content"]),"fn=[v for k,v in list(globals().items()) if callable(v) and not k.startswith('_') and k!='exec'][-1]\nassert [fn(i) for i in range(10)]==[0,1,1,2,3,5,8,13,21,34]; assert fn(50)==12586269025")) + d=ask([{"role":"user","content":"Write a Python function gcd(a, b) that returns the greatest common divisor using Euclid's algorithm. Only the code."}],200); show("gcd",d) + c=d["choices"][0]["message"]["content"]; print(c) + print("gcd tests:",run_tests(code(c),"assert gcd(48,18)==6; assert gcd(17,5)==1; assert gcd(0,9)==9; assert gcd(270,192)==6")) + d=ask([{"role":"user","content":"What is the output of this Python code? print(sum(i*i for i in range(1,11))) Answer with just the number."}],20); show("known",d); print(d["choices"][0]["message"]["content"], "(expected 385)") +if mode=="long": + src="".join(open(f).read() for f in sorted(glob.glob("/tools/*.py"))[:6])[:int(sys.argv[2])] + msgs=[{"role":"user","content":"Here is some Python source:\n\n"+src+"\n\nIn one paragraph, say what this code is for, then write a short Python function that reverses a string."}] + for i in range(2): + d=ask(msgs,256); show(f"long run{i}",d) + print(d["choices"][0]["message"]["content"][:1200]) +if mode=="long2": + fs=sorted(glob.glob("/tools/*.py")) + src="".join(open(f).read() for f in fs[6:12])[:7000] + msgs=[{"role":"user","content":"Here is some Python source:\n\n"+src+"\n\nWrite a Python implementation of a binary search tree class with insert, search, delete and in-order traversal methods, with docstrings."}] + d=ask(msgs,256); show("long2 BST",d) + d2=ask(msgs+[{"role":"assistant","content":d["choices"][0]["message"]["content"]},{"role":"user","content":"Now add a height() method."}],256); show("long2 turn2",d2) + print(d["choices"][0]["message"]["content"][:600]) diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-fib1.json b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-fib1.json new file mode 100644 index 00000000000..8b6e53d22f7 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-fib1.json @@ -0,0 +1 @@ +{"id": "chatcmpl-9a1c5ae9d2084c8f9b34ac8c", "object": "chat.completion", "created": 1791157128, "model": "coder-iq1_m", "choices": [{"index": 0, "message": {"role": "assistant", "content": "```python\ndef fibonacci(n: int) -> int:\n \"\"\"Return the n-th Fibonacci number (0-indexed).\n \n fibonacci(0) = 0\n fibonacci(1) = 1\n fibonacci(2) = 1\n fibonacci(3) = 2\n ...\n \"\"\"\n if n < 0:\n raise ValueError(\"n must be a non-negative integer\")\n if n == 0:\n return 0\n if n == 1:\n return 1\n \n a, b = 0, 1\n for _ in range(2, n + 1):\n a, b = b, a + b\n return b\n```"}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 23, "completion_tokens": 159, "total_tokens": 182, "prompt_tokens_details": {"cached_tokens": 0}}, "timings": {"cache_n": 0, "prompt_n": 23, "prompt_ms": 456.3, "prompt_per_token_ms": 19.839, "prompt_per_second": 50.4, "predicted_n": 159, "predicted_ms": 3009.9, "predicted_per_token_ms": 18.93, "predicted_per_second": 52.8, "draft_n": 138, "draft_n_accepted": 112}} \ No newline at end of file diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long.txt new file mode 100644 index 00000000000..679f45b186b --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long.txt @@ -0,0 +1,8 @@ +[long run0] prompt_n=2129 cache_n=0 prompt 427.9 tok/s | decode 128 tok 40.1 tok/s | drafts 70/98 | wall 8.2s | finish stop +[long run1] prompt_n=7 cache_n=2122 prompt 78.1 tok/s | decode 128 tok 41.3 tok/s | drafts 70/98 | wall 3.2s | finish stop +This code is a comprehensive integration test script for a batched inference engine (likely named "Strata") that verifies the correctness of multi-slot decoding, specifically ensuring that tokens generated in parallel slots match those generated in solo mode, even when long prompts are interleaved with other decoding operations or when requests are continued from checkpoints. The script sets up various scenarios involving different prompt lengths, slot assignments, and continuation logic to validate that the engine's internal state management, expert caching, and token generation remain consistent across different execution paths. + +```python +def reverse_string(s: str) -> str: + return s[::-1] +``` diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long2.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long2.txt new file mode 100644 index 00000000000..590511513b3 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-long2.txt @@ -0,0 +1,26 @@ +[long2 BST] prompt_n=1922 cache_n=0 prompt 425.3 tok/s | decode 256 tok 57.2 tok/s | drafts 186/208 | wall 9.0s | finish length +[long2 turn2] prompt_n=21 cache_n=2178 prompt 61.4 tok/s | decode 256 tok 71.4 tok/s | drafts 209/212 | wall 3.9s | finish length +```python +class BinarySearchTree: + """A binary search tree implementation with insert, search, delete, and in-order traversal.""" + + def __init__(self): + """Initialize an empty binary search tree.""" + self.root = None + + def insert(self, value): + """Insert a value into the binary search tree. + + Args: + value: The value to insert into the tree. + + Returns: + None + """ + if self.root is None: + self.root = _Node(value) + return + + current = self.root + while True: + if value < curr diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-short.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-short.txt new file mode 100644 index 00000000000..4bf19ec8db1 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/c2-short.txt @@ -0,0 +1,35 @@ +[fib run0] prompt_n=7 cache_n=16 prompt 74.3 tok/s | decode 159 tok 56.6 tok/s | drafts 114/133 | wall 2.9s | finish stop +[fib run1] prompt_n=7 cache_n=16 prompt 80.1 tok/s | decode 159 tok 56.6 tok/s | drafts 114/133 | wall 2.9s | finish stop +```python +def fibonacci(n: int) -> int: + """Return the n-th Fibonacci number (0-indexed). + + fibonacci(0) = 0 + fibonacci(1) = 1 + fibonacci(2) = 1 + fibonacci(3) = 2 + ... + """ + if n < 0: + raise ValueError("n must be a non-negative integer") + if n == 0: + return 0 + if n == 1: + return 1 + + a, b = 0, 1 + for _ in range(2, n + 1): + a, b = b, a + b + return b +``` +fib tests: PASS +[gcd] prompt_n=37 cache_n=0 prompt 71.7 tok/s | decode 32 tok 60.8 tok/s | drafts 23/24 | wall 1.0s | finish stop +```python +def gcd(a, b): + while b: + a, b = b, a % b + return a +``` +gcd tests: PASS +[known] prompt_n=42 cache_n=0 prompt 69.5 tok/s | decode 4 tok 25.1 tok/s | drafts 4/6 | wall 0.8s | finish stop +385 (expected 385) diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.engine.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.engine.txt new file mode 100644 index 00000000000..a8651328d75 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.engine.txt @@ -0,0 +1,66 @@ +strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 24) +strata generate: PCIe probe: 7.0 GB/s host->device (best of 7.0 7.0 7.0 7.0) -> pcie_frac 0.19 (default 0.55) +strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows) +strata generate: native pack: /packs/coder-iq1_m experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB) +strata generate: layer split: CUDA0 loads the dense weights of layers 0-23 only +strata generate: 741 MiB of weights loaded from /packs/coder-iq1_m (686 canonical tensors skipped: served natively) +strata generate: 150 native projection matrices, 988.19 MiB of weights +strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off) +strata generate: layer split: CUDA1 loads the dense weights of layers 24-47 only +strata generate: layer split: CUDA1 PCIe probe 7.0 GB/s (best of 7.0 7.0 7.0 7.0) -> pcie_frac 0.19 +strata generate: layer split: CUDA1 holds its weights; 22.04 GiB free (its session follows the split search) +strata generate: profile data/expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots +strata generate: PLE on, table 320001536 rows of /models/coder-iq1_m/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf +strata generate: layer split: CUDA1 holds its weights, session [24, 48) and the head; 21.43 GiB free +strata mtp: draft layer loaded, 802 MiB of VRAM (experts 675, dense 111), files read in 0.44 s (1769 MiB/s) +strata generate: GPU 0: Intel(R) Arc(TM) Pro B60 Graphics, compute capability 20.1 +strata generate: WARNING: this engine was compiled with CUDA 12.8 headers but loaded a CUDA 0.2 runtime (libcudart): GPU properties can read wrong and some kernels go unused. Rebuild it against one toolkit (cmake -DCUDAToolkit_ROOT=, with its libcudart.so present) (#542) +strata generate: experts streamed from the GGUF on demand (--stream-experts): no host arena; every expert must fit the VRAM cache +strata generate: expert cache auto: 21.95 GiB free, 1536 MiB reserved (+0 MiB for the draft head) -> 6144 slots +strata generate: expert cache 6144 slots, 11.07 GiB of VRAM; policy is +strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU, + so a reply can differ slightly from a run without the cache (same quality: + bench/results/2026-09-27-cache-parity). + PROFILE, ranked by routing frequency, no eviction. +strata generate: profile fill: 11.07 GiB in 1.9 s (6.27 GB/s, pipelined) +strata generate: pre-filled 6144 of 6144 slots from the profile; slot 0 verified +strata generate: 3024 of 6144 experts missing from VRAM mirrored in pinned host memory (6.00 GiB, 1.0 s); the GPU reads them over PCIe +strata generate: WARNING: 3120 experts are neither in VRAM nor mirrored; with STRATA_VERIFY_NO_HOST the device plan cannot run them and their layers' windows fall back slowly - raise STRATA_MIRROR_MIB or the free RAM, or lower --max-context +strata generate: layer split, CUDA1: 20.65 GiB free of 23.91, room for experts 17.33 GiB +strata generate: layer split: CUDA1 runs layers 24-47, expert cache 6144 slots (12.35 GiB), 6144 of its 6144 profiled pairs; slot 0 verified +strata generate: layer split: CUDA0 runs layers 0-23 +strata generate: R4 hit path ON - resident experts are computed on the GPU +strata generate: 5 expert-pool workers + the host thread +strata generate: session is up (engine 0.1.39-sycl) +strata generate: token graph hit path: 12288 resident experts, decided on the device +strata serve: layer split: 100% of the experts resident, the prompt path's streamed ring 96 slots +strata serve: the prompt path allocates its own buffers (too few cache slots to borrow) +strata verify: window up to 6 tokens, 74.0 MiB of device buffers +strata serve: layer split: layers 0-23 (CUDA0), 24-47 (CUDA1), one hand-off per window +strata verify: window up to 6 tokens, 74.0 MiB of device buffers +strata mtp: draft head over 106299 tokens (212.9 MiB) +strata serve: 9890 MiB of VRAM free with everything loaded +strata verify: captured the 6-token window (upload , sync ) +strata verify: captured the 6-token window (upload , sync ) +strata verify: captured the 4-token window (upload , sync ) +strata verify: captured the 4-token window (upload , sync ) +strata verify: captured the 1-token window (upload , sync ) +strata verify: captured the 1-token window (upload , sync ) +strata verify: captured the 2-token window (upload , sync ) +strata verify: captured the 2-token window (upload , sync ) +strata serve: prompt 23 tokens = 0 reused + 23 read in 456 ms (50.4 tok/s), 159 generated in 3010 ms (52.8 tok/s), drafts accepted 112 of 138, 1 checkpoints +strata serve: suffix drafts: 1 windows, 1 of 5 drafts accepted +strata serve: prompt 23 tokens = 16 reused + 7 read in 94 ms (74.3 tok/s), 159 generated in 2810 ms (56.6 tok/s), drafts accepted 114 of 133, 1 checkpoints +strata serve: prompt 23 tokens = 16 reused + 7 read in 87 ms (80.1 tok/s), 159 generated in 2808 ms (56.6 tok/s), drafts accepted 114 of 133, 1 checkpoints +strata serve: prompt 37 tokens = 0 reused + 37 read in 516 ms (71.7 tok/s), 32 generated in 526 ms (60.8 tok/s), drafts accepted 23 of 24, 1 checkpoints +strata verify: captured the 5-token window (upload , sync ) +strata verify: captured the 5-token window (upload , sync ) +strata serve: prompt 42 tokens = 0 reused + 42 read in 604 ms (69.5 tok/s), 4 generated in 160 ms (25.1 tok/s), drafts accepted 4 of 6, 1 checkpoints +strata verify: captured the 3-token window (upload , sync ) +strata verify: captured the 3-token window (upload , sync ) +strata serve: prompt 2129 tokens = 0 reused + 2129 read in 4975 ms (427.9 tok/s), 128 generated in 3189 ms (40.1 tok/s), drafts accepted 70 of 98, 1 checkpoints +strata serve: prompt 2129 tokens = 2122 reused + 7 read in 90 ms (78.1 tok/s), 128 generated in 3101 ms (41.3 tok/s), drafts accepted 70 of 98, 1 checkpoints +strata serve: prompt 1922 tokens = 0 reused + 1922 read in 4519 ms (425.3 tok/s), 256 generated in 4475 ms (57.2 tok/s), drafts accepted 186 of 208, 1 checkpoints +strata serve: suffix drafts: 1 windows, 2 of 5 drafts accepted +strata serve: prompt 2199 tokens = 2178 reused + 21 read in 342 ms (61.4 tok/s), 256 generated in 3587 ms (71.4 tok/s), drafts accepted 209 of 212, 2 checkpoints +strata serve: suffix drafts: 37 windows, 185 of 185 drafts accepted diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.json b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.json new file mode 100644 index 00000000000..dec1a68c1e0 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/coder2.json @@ -0,0 +1,47 @@ +{ + "engine": "strata", + "exe": "/strata-native.sh", + "cwd": "", + "tokenizer": "/packs/coder-iq1_m/tokenizer", + "model_name": "coder-iq1_m", + "backend": "sycl", + "log": "/logs/w2/coder2.engine.log", + "args": [ + "--pack", + "/packs/coder-iq1_m", + "--native", + "/models/coder-iq1_m/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf", + "--ple-gguf", + "/models/coder-iq1_m/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf", + "--expert-profile", + "data/expert-profile-coder.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--spec", + "4", + "--spec-min-p", + "0.5", + "--mtp", + "/mtp/rt", + "--max-context", + "8192", + "--kv", + "int8", + "--vram-reserve-mib", + "1536", + "--no-prefill-borrow", + "--layer-split", + "24" + ], + "env": { + "ONEAPI_DEVICE_SELECTOR": "level_zero:0,1", + "SYCL_CACHE_PERSISTENT": "0", + "STRATA_MIRROR_MIB": "6144", + "STRATA_VERIFY_DEVICE_PLAN": "1", + "STRATA_STAGE_TRIM": "1", + "STRATA_VERIFY_NO_HOST": "1" + } +} \ No newline at end of file diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long.txt new file mode 100644 index 00000000000..c9aedcee380 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long.txt @@ -0,0 +1,8 @@ +[long run0] prompt_n=2129 cache_n=0 prompt 436.4 tok/s | decode 104 tok 38.8 tok/s | drafts 56/87 | wall 7.6s | finish stop +[long run1] prompt_n=7 cache_n=2122 prompt 79.3 tok/s | decode 104 tok 40.0 tok/s | drafts 56/87 | wall 2.7s | finish stop +This code serves as a comprehensive test harness for a batch inference engine, specifically designed to verify that token generation remains consistent when multiple prompts are processed concurrently in separate slots. It compares the output tokens from parallel execution against those from sequential "solo" runs to ensure that interleaving, checkpointing, and resource sharing (such as expert cache borrowing) do not introduce drift or errors in the model's responses. + +```python +def reverse_string(s: str) -> str: + return s[::-1] +``` diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long2.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long2.txt new file mode 100644 index 00000000000..943b358ef4b --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-long2.txt @@ -0,0 +1,23 @@ +[long2 BST] prompt_n=1922 cache_n=0 prompt 445.1 tok/s | decode 256 tok 55.5 tok/s | drafts 183/213 | wall 9.0s | finish length +[long2 turn2] prompt_n=284 cache_n=1915 prompt 177.0 tok/s | decode 256 tok 72.5 tok/s | drafts 211/217 | wall 5.1s | finish length +```python +class BinarySearchTree: + """A binary search tree implementation with insert, search, delete, and in-order traversal.""" + + class Node: + """Internal node of the binary search tree.""" + + def __init__(self, value): + """Initialize a node with a value and no children.""" + self.value = value + self.left = None + self.right = None + + def __init__(self): + """Initialize an empty binary search tree.""" + self.root = None + + def insert(self, value): + """Insert a value into the binary search tree. + + Args: diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-short.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-short.txt new file mode 100644 index 00000000000..e6c92f98d7b --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/i2-short.txt @@ -0,0 +1,27 @@ +[fib run0] prompt_n=23 cache_n=0 prompt 50.4 tok/s | decode 112 tok 58.6 tok/s | drafts 84/88 | wall 2.4s | finish stop +[fib run1] prompt_n=7 cache_n=16 prompt 81.7 tok/s | decode 112 tok 61.3 tok/s | drafts 84/89 | wall 1.9s | finish stop +```python +def fibonacci(n): + """Return the n-th Fibonacci number (0-indexed).""" + if n < 0: + raise ValueError("n must be a non-negative integer") + if n == 0: + return 0 + if n == 1: + return 1 + a, b = 0, 1 + for _ in range(2, n + 1): + a, b = b, a + b + return b +``` +fib tests: PASS +[gcd] prompt_n=37 cache_n=0 prompt 73.3 tok/s | decode 32 tok 61.8 tok/s | drafts 23/24 | wall 1.0s | finish stop +```python +def gcd(a, b): + while b: + a, b = b, a % b + return a +``` +gcd tests: PASS +[known] prompt_n=42 cache_n=0 prompt 71.1 tok/s | decode 4 tok 25.4 tok/s | drafts 4/6 | wall 0.7s | finish stop +385 (expected 385) diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.engine.txt b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.engine.txt new file mode 100644 index 00000000000..6dcb24c0b74 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.engine.txt @@ -0,0 +1,69 @@ +strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 25) +strata generate: PCIe probe: 7.0 GB/s host->device (best of 7.0 7.0 7.0 7.0) -> pcie_frac 0.19 (default 0.55) +strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows) +strata generate: native pack: /packs/qwen-iq2_xs experts (largest blob 1.51 MB), token embedding IQ4_XS in mapped host memory (322 MiB) +strata generate: layer split: CUDA0 loads the dense weights of layers 0-24 only +strata generate: 799 MiB of weights loaded from /packs/qwen-iq2_xs (670 canonical tensors skipped: served natively) +strata generate: 156 native projection matrices, 766.85 MiB of weights +strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off) +strata generate: layer split: CUDA1 loads the dense weights of layers 25-47 only +strata generate: layer split: CUDA1 PCIe probe 7.0 GB/s (best of 7.0 7.0 7.0 7.0) -> pcie_frac 0.19 +strata generate: layer split: CUDA1 holds its weights; 22.36 GiB free (its session follows the split search) +strata generate: profile data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots +strata generate: PLE on, table 320001536 rows of /models/qwen-iq2_xs/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf +strata generate: layer split: CUDA1 holds its weights, session [25, 48) and the head; 21.93 GiB free +strata mtp: draft layer loaded, 802 MiB of VRAM (experts 675, dense 111), files read in 0.75 s (1053 MiB/s) +strata generate: GPU 0: Intel(R) Arc(TM) Pro B60 Graphics, compute capability 20.1 +strata generate: WARNING: this engine was compiled with CUDA 12.8 headers but loaded a CUDA 0.2 runtime (libcudart): GPU properties can read wrong and some kernels go unused. Rebuild it against one toolkit (cmake -DCUDAToolkit_ROOT=, with its libcudart.so present) (#542) +strata generate: experts streamed from the GGUF on demand (--stream-experts): no host arena; every expert must fit the VRAM cache +strata generate: expert cache auto: 22.11 GiB free, 1536 MiB reserved (+0 MiB for the draft head) -> 12800 slots +strata generate: expert cache 12800 slots, 16.91 GiB of VRAM; policy is +strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU, + so a reply can differ slightly from a run without the cache (same quality: + bench/results/2026-09-27-cache-parity). + PROFILE, ranked by routing frequency, no eviction. +strata generate: profile fill: 16.91 GiB in 3.8 s (4.81 GB/s, pipelined) +strata generate: pre-filled 12800 of 12800 slots from the profile; slot 0 verified +strata generate: 4473 of 11776 experts missing from VRAM mirrored in pinned host memory (6.00 GiB, 1.1 s); the GPU reads them over PCIe +strata generate: WARNING: 7303 experts are neither in VRAM nor mirrored; with STRATA_VERIFY_NO_HOST the device plan cannot run them and their layers' windows fall back slowly - raise STRATA_MIRROR_MIB or the free RAM, or lower --max-context +strata generate: layer split, CUDA1: 21.14 GiB free of 23.91, room for experts 17.93 GiB +strata generate: layer split: CUDA1 runs layers 25-47, expert cache 11776 slots (16.11 GiB), 11776 of its 11776 profiled pairs; slot 0 verified +strata generate: layer split: CUDA0 runs layers 0-24 +strata generate: R4 hit path ON - resident experts are computed on the GPU +strata generate: 5 expert-pool workers + the host thread +strata generate: session is up (engine 0.1.39-sycl) +strata generate: token graph hit path: 24576 resident experts, decided on the device +strata serve: layer split: 100% of the experts resident, the prompt path's streamed ring 96 slots +strata serve: the prompt path allocates its own buffers (too few cache slots to borrow) +strata verify: window up to 6 tokens, 56.4 MiB of device buffers +strata serve: layer split: layers 0-24 (CUDA0), 25-47 (CUDA1), one hand-off per window +strata verify: window up to 6 tokens, 56.4 MiB of device buffers +strata mtp: draft head over 106299 tokens (137.9 MiB) +strata serve: 4235 MiB of VRAM free with everything loaded +strata verify: captured the 6-token window (upload , sync ) +strata verify: captured the 6-token window (upload , sync ) +strata verify: captured the 4-token window (upload , sync ) +strata verify: captured the 4-token window (upload , sync ) +strata verify: captured the 1-token window (upload , sync ) +strata verify: captured the 1-token window (upload , sync ) +strata verify: captured the 3-token window (upload , sync ) +strata verify: captured the 3-token window (upload , sync ) +strata serve: prompt 23 tokens = 0 reused + 23 read in 456 ms (50.4 tok/s), 112 generated in 1910 ms (58.6 tok/s), drafts accepted 84 of 88, 1 checkpoints +strata serve: prompt 23 tokens = 16 reused + 7 read in 86 ms (81.7 tok/s), 112 generated in 1827 ms (61.3 tok/s), drafts accepted 84 of 89, 1 checkpoints +strata serve: suffix drafts: 1 windows, 1 of 3 drafts accepted +strata serve: prompt 37 tokens = 0 reused + 37 read in 505 ms (73.3 tok/s), 32 generated in 518 ms (61.8 tok/s), drafts accepted 23 of 24, 1 checkpoints +strata verify: captured the 5-token window (upload , sync ) +strata verify: captured the 5-token window (upload , sync ) +strata serve: prompt 42 tokens = 0 reused + 42 read in 591 ms (71.1 tok/s), 4 generated in 157 ms (25.4 tok/s), drafts accepted 4 of 6, 1 checkpoints +strata verify: captured the 2-token window (upload , sync ) +strata verify: captured the 2-token window (upload , sync ) +strata serve: prompt 2129 tokens = 0 reused + 2129 read in 4878 ms (436.4 tok/s), 104 generated in 2680 ms (38.8 tok/s), drafts accepted 56 of 87, 1 checkpoints +strata serve: prompt 2129 tokens = 2122 reused + 7 read in 88 ms (79.2 tok/s), 104 generated in 2602 ms (40.0 tok/s), drafts accepted 56 of 87, 1 checkpoints +strata serve: prompt 1922 tokens = 0 reused + 1922 read in 4318 ms (445.1 tok/s), 256 generated in 4616 ms (55.5 tok/s), drafts accepted 183 of 213, 1 checkpoints +strata serve: suffix drafts: 1 windows, 1 of 5 drafts accepted +strata serve: prompt 2199 tokens = 1915 reused + 284 read in 1604 ms (177.0 tok/s), 256 generated in 3532 ms (72.5 tok/s), drafts accepted 211 of 217, 2 checkpoints +strata serve: suffix drafts: 41 windows, 205 of 205 drafts accepted +strata serve: prompt 30 tokens = 0 reused + 30 read in 454 ms (66.1 tok/s), 11 generated in 255 ms (43.2 tok/s), drafts accepted 7 of 9, 1 checkpoints +strata serve: prompt 70 tokens = 41 reused + 29 read in 412 ms (70.3 tok/s), 3 generated in 97 ms (30.9 tok/s), drafts accepted 3 of 3, 2 checkpoints +strata serve: prompt 108 tokens = 74 reused + 34 read in 479 ms (70.9 tok/s), 14 generated in 307 ms (45.6 tok/s), drafts accepted 8 of 11, 3 checkpoints +strata serve: prompt 33 tokens = 0 reused + 33 read in 487 ms (67.8 tok/s), 39 generated in 813 ms (47.9 tok/s), drafts accepted 28 of 38, 1 checkpoints diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.json b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.json new file mode 100644 index 00000000000..ae1884edfe1 --- /dev/null +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/data/iq2b.json @@ -0,0 +1,47 @@ +{ + "engine": "strata", + "exe": "/strata-native.sh", + "cwd": "", + "tokenizer": "/packs/qwen-iq2_xs/tokenizer", + "model_name": "qwen3.8-flash-next-iq2_xs", + "backend": "sycl", + "log": "/logs/w2/iq2b.engine.log", + "args": [ + "--pack", + "/packs/qwen-iq2_xs", + "--native", + "/models/qwen-iq2_xs/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", + "--ple-gguf", + "/models/qwen-iq2_xs/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf", + "--expert-profile", + "data/expert-profile.bin", + "--expert-cache", + "auto", + "--stream-experts", + "--prefill", + "auto", + "--spec", + "4", + "--spec-min-p", + "0.5", + "--mtp", + "/mtp/rt", + "--max-context", + "8192", + "--kv", + "int8", + "--vram-reserve-mib", + "1536", + "--no-prefill-borrow", + "--layer-split", + "25" + ], + "env": { + "ONEAPI_DEVICE_SELECTOR": "level_zero:0,1", + "SYCL_CACHE_PERSISTENT": "0", + "STRATA_MIRROR_MIB": "6144", + "STRATA_VERIFY_DEVICE_PLAN": "1", + "STRATA_STAGE_TRIM": "1", + "STRATA_VERIFY_NO_HOST": "1" + } +} \ No newline at end of file diff --git a/docs/COMMUNITY_BENCHMARKS.md b/docs/COMMUNITY_BENCHMARKS.md index e6903faf401..f4198f17c6d 100644 --- a/docs/COMMUNITY_BENCHMARKS.md +++ b/docs/COMMUNITY_BENCHMARKS.md @@ -18,6 +18,9 @@ and their limits. Report what you actually measured and label estimates separate - [2026-10-03: 2x AMD Instinct MI50 16 GB (gfx906), Xeon E5-2666 v3, 32 GB RAM](../bench/results/2026-10-03-community-2x-mi50/README.md): the gfx906 build (#638) with #639 and #640, Coder IQ1_M, 131,072-token context, layer split across both cards; three runs each at 4,096, 32,768, and 128,000 prompt tokens, plus six recall checks. +- [2026-10-04: 2x Intel Arc Pro B60 24 GB, Ryzen 5 5600, 62 GB RAM](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md): + the SYCL port at `6f32ec0` with two fixes, Coder IQ1_M and Flash-Next IQ2_XS, 8,192-token context, layer split across both cards; + short prompts and 2K-token prompts, plus one run on a single B60. Intel's SYCL engine, no 4,096/32,768/128,000-token sweep. ## What to record diff --git a/docs/INTEL_ARC.md b/docs/INTEL_ARC.md index 7b6acfeb637..b08481fe33e 100644 --- a/docs/INTEL_ARC.md +++ b/docs/INTEL_ARC.md @@ -19,12 +19,18 @@ There is **no ready-made Intel engine** in the release zips. You build it from s | Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt (with the `stage_room` fix that is now in 0.1.39) | #423 | | Community | Arc Pro B50 16 GB | Coder IQ1_M ~23 tok/s, IQ2_XS ~25-27 tok/s, up to 128K | #423 | | Community | Arc B580 12 GB, WSL2 | IQ2_XS ~21 tok/s, Coder ~15 tok/s; **device loss also seen** | #423 | +| Community | 2x Arc Pro B60 24 GB, `--layer-split` | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [bench/results/2026-10-04-community-2x-arc-pro-b60](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) | +| Community | one Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode (needs the ring-wait fix from the same report) | same | | Strata maintainers | no Arc | compile check and kernel tests on a CPU device only (below) | this release | The 0.1.39 port re-migrates 0.1.38's port onto the 0.1.39 engine sources (the #606 NaN fix, the #649 verify trace, the new prompt paths). It compiles and its kernel tests run, but **nobody has run the 0.1.39 port on an Arc yet**. The numbers in the table were measured on earlier versions. +The two B60 rows ran `6f32ec0` plus two small `sycl/` fixes (the compile fix and the ring-wait fix), AOT `bmg-g21`, on Ubuntu 24.04 with +`xe`, Level Zero V2, NEO 26.09.37435.12 and oneAPI 2026.1.1, without Docker. Host: Ryzen 5 5600, 62 GB RAM, PCIe 3.0 x8 per card. +Full flags, per-request timings and engine logs are in the report linked in the table. + ## What was tested here (0.1.39) - **Compiles:** Ubuntu 24.04 (WSL2), Intel oneAPI DPC++ 2026.1.1 + oneMKL 2026.1. The whole `sycl/` project @@ -108,6 +114,16 @@ Things that matter on an Arc (details in INTEL.md): - `SYCL_CACHE_PERSISTENT=0`: the persistent JIT cache crashed on Xe2 during the first compile. - Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` now passes the variable through) and `--layer-split`. +- **Arc Pro B60:** the PCI id is `8086:e211` (`lspci -nn`, the kernel's `xe` id list files it with the BMG-G21 cards), `sycl-ls` prints + `Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, and `ocloc ids bmg-g21` prints 20.1.0, so `-DSTRATA_SYCL_AOT=bmg-g21` is the right target. + `setup_intel.py` knows `e211` (and still lists `e221`). On a `--layer-split` the startup line "N experts are neither in VRAM nor mirrored" + counts the other card's layers; the lines `100% of the experts resident` that follow are the ones to read. Set `STRATA_MIRROR_MIB=0` there, + or the pinned mirror is allocated and not used. +- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at `OCL_ICD_FILENAMES: unbound variable` + (oneAPI 2026.1.1). Source it before `set -u`, or run `set +u` around it. +- **Without Docker:** `sycl/serve/strata-sycl.sh` needs Docker. Natively, `source setvars.sh`, export `SYCL_CACHE_PERSISTENT=0`, + `ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py` with an `exe` that does this. + A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU device. - `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh`) is only valid when every expert is in VRAM. On smaller cards that path is the one that has hung, and #667 found the likely reason: the GPU does not see the CPU's flag stores without a system fence. diff --git a/sycl/setup_intel.py b/sycl/setup_intel.py index 0b1e2f05899..9e615bc10b0 100644 --- a/sycl/setup_intel.py +++ b/sycl/setup_intel.py @@ -41,7 +41,8 @@ # Battlemage / Alchemist PCI device ids -> (name, VRAM GB). lspci's database lags new cards (an Arc Pro B70 reads # "Intel Corporation Device [8086:e223]"), so the sysfs id is the reliable signal and the name table is ours. -INTEL_ARC = {"e223": ("Arc Pro B70", 32.0), "e221": ("Arc Pro B60", 24.0), "e20b": ("Arc B580", 12.0), +INTEL_ARC = {"e223": ("Arc Pro B70", 32.0), "e221": ("Arc Pro B60", 24.0), "e211": ("Arc Pro B60", 24.0), + "e20b": ("Arc B580", 12.0), "e20c": ("Arc B570", 10.0), "e212": ("Arc B50", 16.0), "56a0": ("Arc A770", 16.0), "56a1": ("Arc A750", 8.0), "56a2": ("Arc A580", 8.0), "56a5": ("Arc A380", 6.0), "56a6": ("Arc A310", 4.0), "5690": ("Arc A770M", 16.0)} diff --git a/tools/test_setup_sycl.py b/tools/test_setup_sycl.py index 2ff02b64a3a..49badcab2ca 100644 --- a/tools/test_setup_sycl.py +++ b/tools/test_setup_sycl.py @@ -71,5 +71,26 @@ def test_sycl_is_off_by_default_and_exclusive(self): self.assertIn("add_subdirectory(sycl)", src) +class IntelIds(unittest.TestCase): + """sycl/setup_intel.py names a card by its PCI device id. The Arc Pro B60s measured on 2x B60 (`lspci -nn` 8086:e211, + bmg-g21) must not fall through to "Intel GPU e211 (xe)" with a guessed VRAM size.""" + + @staticmethod + def table(): + import importlib.util + spec = importlib.util.spec_from_file_location("setup_intel", ROOT / "sycl" / "setup_intel.py") + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod.INTEL_ARC + + def test_b60_e211_is_named_with_its_vram(self): + self.assertEqual(self.table()["e211"], ("Arc Pro B60", 24.0)) + + def test_the_ids_that_were_listed_before_are_kept(self): + t = self.table() + self.assertEqual(t["e223"][0], "Arc Pro B70") + self.assertEqual(t["e221"][0], "Arc Pro B60") + + if __name__ == "__main__": unittest.main() From 08593bc6a1abee044c73c9056283ee89dea4aa7d Mon Sep 17 00:00:00 2001 From: LocalXPU <337401568+LocalXPU@users.noreply.github.com> Date: Sun, 4 Oct 2026 19:54:32 -0500 Subject: [PATCH 2/3] docs: the host has 64 GB of RAM (62 GiB usable), not 62 GB --- bench/results/2026-10-04-community-2x-arc-pro-b60/README.md | 2 +- docs/COMMUNITY_BENCHMARKS.md | 2 +- docs/INTEL_ARC.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md b/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md index 1c57d1cd00f..2935e016974 100644 --- a/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md +++ b/bench/results/2026-10-04-community-2x-arc-pro-b60/README.md @@ -4,7 +4,7 @@ Measured on 2026-10-04 by LocalXPU. This is the SYCL port (`sycl/`) on two Arc P ## Hardware and software -- 2x Intel Arc Pro B60 24 GB (`8086:e211`, 456 GB/s each), PCIe 3.0 x8 per card; Ryzen 5 5600; 62 GB DDR4; storage type not recorded +- 2x Intel Arc Pro B60 24 GB (`8086:e211`, 456 GB/s each), PCIe 3.0 x8 per card; Ryzen 5 5600; 64 GB DDR4; storage type not recorded - Ubuntu 24.04, kernel 6.17 (`xe` driver), Level Zero V2, compute runtime NEO 26.09.37435.12, oneAPI DPC++ 2026.1.1, oneMKL 2026.1.0 - Strata `6f32ec0` plus two `sycl/` fixes (the build fix for #559/#626 and the `cudaStreamQuery` fix in the ring wait); engine reports `0.1.39-sycl`; AOT `-DSTRATA_SYCL_AOT=bmg-g21`; source build, no Docker - Nothing else ran on the GPUs. The kernel log was watched for `xe` resets during every run: none. diff --git a/docs/COMMUNITY_BENCHMARKS.md b/docs/COMMUNITY_BENCHMARKS.md index f4198f17c6d..adb9f736e14 100644 --- a/docs/COMMUNITY_BENCHMARKS.md +++ b/docs/COMMUNITY_BENCHMARKS.md @@ -18,7 +18,7 @@ and their limits. Report what you actually measured and label estimates separate - [2026-10-03: 2x AMD Instinct MI50 16 GB (gfx906), Xeon E5-2666 v3, 32 GB RAM](../bench/results/2026-10-03-community-2x-mi50/README.md): the gfx906 build (#638) with #639 and #640, Coder IQ1_M, 131,072-token context, layer split across both cards; three runs each at 4,096, 32,768, and 128,000 prompt tokens, plus six recall checks. -- [2026-10-04: 2x Intel Arc Pro B60 24 GB, Ryzen 5 5600, 62 GB RAM](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md): +- [2026-10-04: 2x Intel Arc Pro B60 24 GB, Ryzen 5 5600, 64 GB RAM](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md): the SYCL port at `6f32ec0` with two fixes, Coder IQ1_M and Flash-Next IQ2_XS, 8,192-token context, layer split across both cards; short prompts and 2K-token prompts, plus one run on a single B60. Intel's SYCL engine, no 4,096/32,768/128,000-token sweep. diff --git a/docs/INTEL_ARC.md b/docs/INTEL_ARC.md index b08481fe33e..0a1b4f4e165 100644 --- a/docs/INTEL_ARC.md +++ b/docs/INTEL_ARC.md @@ -28,7 +28,7 @@ trace, the new prompt paths). It compiles and its kernel tests run, but **nobody yet**. The numbers in the table were measured on earlier versions. The two B60 rows ran `6f32ec0` plus two small `sycl/` fixes (the compile fix and the ring-wait fix), AOT `bmg-g21`, on Ubuntu 24.04 with -`xe`, Level Zero V2, NEO 26.09.37435.12 and oneAPI 2026.1.1, without Docker. Host: Ryzen 5 5600, 62 GB RAM, PCIe 3.0 x8 per card. +`xe`, Level Zero V2, NEO 26.09.37435.12 and oneAPI 2026.1.1, without Docker. Host: Ryzen 5 5600, 64 GB RAM, PCIe 3.0 x8 per card. Full flags, per-request timings and engine logs are in the report linked in the table. ## What was tested here (0.1.39) From 2f188172e4223db289d288f80a526ade0f10be87 Mon Sep 17 00:00:00 2001 From: LocalXPU <337401568+LocalXPU@users.noreply.github.com> Date: Sun, 4 Oct 2026 22:35:40 -0500 Subject: [PATCH 3/3] docs: e221 is also an Arc Pro B60 id (seen on another B60 pair) --- docs/INTEL_ARC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/INTEL_ARC.md b/docs/INTEL_ARC.md index 0a1b4f4e165..bfaddde9fa7 100644 --- a/docs/INTEL_ARC.md +++ b/docs/INTEL_ARC.md @@ -116,7 +116,7 @@ Things that matter on an Arc (details in INTEL.md): variable through) and `--layer-split`. - **Arc Pro B60:** the PCI id is `8086:e211` (`lspci -nn`, the kernel's `xe` id list files it with the BMG-G21 cards), `sycl-ls` prints `Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, and `ocloc ids bmg-g21` prints 20.1.0, so `-DSTRATA_SYCL_AOT=bmg-g21` is the right target. - `setup_intel.py` knows `e211` (and still lists `e221`). On a `--layer-split` the startup line "N experts are neither in VRAM nor mirrored" + `setup_intel.py` knows both B60 ids, `e211` and `e221` (both seen on B60 cards). On a `--layer-split` the startup line "N experts are neither in VRAM nor mirrored" counts the other card's layers; the lines `100% of the experts resident` that follow are the ones to read. Set `STRATA_MIRROR_MIB=0` there, or the pinned mirror is allocated and not used. - **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at `OCL_ICD_FILENAMES: unbound variable`