Skip to content

[SYCL / Intel Arc] 2x Arc Pro B70 --layer-split auto deadlocks at startup after the host-mirror fill #1054

New issue
New issue
Closed as not planned
Closed as not planned
[SYCL / Intel Arc] 2x Arc Pro B70 --layer-split auto deadlocks at startup after the host-mirror fill#1054

Description

@djbrettb
djbrettb
opened on Oct 6, 2026

Summary. On two Arc Pro B70s, Flash-Next IQ3_XXS and IQ3_S with --layer-split auto load, fill the first
card's expert cache, mirror the remaining experts into pinned host memory, and then never reach "session is up".
One host thread spins at 100 % CPU, the second card reports 100 % busy at ~88 W (a spin, not compute: real
compute on this card draws 200+ W), and the first card idles at 0 MHz. The server keeps printing
still starting (...). It happens with and without PR #866 and with STRATA_VERIFY_NO_HOST=0 or =1.

The same build runs IQ2_XS on one B70 without any problem: 79 tok/s short decode, a 12-minute soak with
156/156 requests and 0 xe engine resets.

What looks inconsistent in the log

The split planner says both caches together hold every profiled pair. But the session's expert cache is sized
only on CUDA0, and the rest goes to the host mirror, while CUDA1 still has ~28 GiB free:

strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: PCIe probe: 18.2 GB/s host->device (best of 18.0 18.2 18.1 18.1) -> pcie_frac 0.50 (default 0.55)
strata generate: layer split: CUDA1 PCIe probe 21.1 GB/s (best of 21.1 19.2 18.0 17.9) -> pcie_frac 0.55
strata generate: layer split: CUDA1 holds its weights; 28.57 GiB free (its session follows the split search)
strata generate: layer split auto: CUDA0 32 SMs at 2.80 GHz -> 0.81 ms per layer, 24.87 GiB free before its session carve
strata generate: layer split auto: CUDA1 32 SMs at 2.80 GHz -> 0.81 ms per layer, 23.81 GiB free before its session carve
strata generate: layer split auto: K=29 - predicted 38.9 ms per decode window; the caches hold 24576 of 24576 profiled pairs (~100.0% of the routed mass)
strata generate: layer split: CUDA1 holds its weights, session [29, 48) and the head; 27.92 GiB free
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111), files read in 0.39 s (2005 MiB/s)
strata generate: expert cache auto: 28.22 GiB free, 2048 MiB reserved (+0 MiB for the draft head) -> 11401 slots
strata generate: expert cache 14848 slots, 21.74 GiB of VRAM; policy is
strata generate: pre-filled 14848 of 14848 slots from the profile; slot 0 verified
strata generate: 9728 of 9728 experts missing from VRAM mirrored in pinned host memory (18.23 GiB, 2.7 s); the GPU reads them over PCIe
(nothing further; the server prints "still starting (N s)" until killed)

IQ3_S behaves the same way: K=26, 13,312 slots / 24.19 GiB on CUDA0, and 11,264 experts (22.64 GiB) mirrored.

--check (sycl/setup_intel.py --check) reports IQ3_XXS and IQ3_S as "fits in VRAM across the 2 cards (layer
split)". So we expected no host mirror at all.

State while stuck

  • Engine process: 78 threads. 77 sleep in futex_do_wait; one runs at 100 % CPU.
  • Card 0 (0000:03:00.0): 0 % busy, 0 MHz. Card 1 (0000:08:00.0): 100 % busy (gtidle residency), 2.8 GHz,
    ~88 W.
  • No xe engine resets or AER errors while it hangs.
  • Killing the container produced one reset on card 1:
    xe 0000:08:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x289
    followed by Fault response: Unsuccessful -ENOENT. The card was healthy afterwards; llama.cpp loaded and
    served on it.

Our guess, which we have not verified in the source: with the layer split, the second stage waits on a host-mapped
flag/doorbell that the host writes while CUDA1's graph is running. docs/INTEL.md notes that
host-to-device visibility during a kernel is unreliable on this platform. The host then waits for CUDA1, so the
two never meet.

What we tried

variant result
maxfridbe/Strata_B70@b70 8dd4aaa (AOT bmg-g31), IQ3_S and IQ3_XXS, --layer-split auto, --vram-reserve-mib 2048 hang (both)
same + PR #866's change applied by its own fixups.py regex (2 sites) hang
same + PR #866 + STRATA_VERIFY_NO_HOST=0 (made overridable in strata-sycl.sh) hang
IQ2_XS on ONE card (no split), --vram-reserve-mib 3072 works: 79 tok/s short, 66 tok/s at 2.5K prompt, 12-min soak clean
llama.cpp b11151 SYCL, same IQ3_XXS GGUF, -sm layer across both cards, --override-tensor per_layer_token_embd=CPU works: ~25 tok/s, 10-min soak clean

Not tried yet: --layer-split K with an explicit K, --split-device, STRATA_VERIFY_DEVICE_PLAN=0,
STRATA_WARM_GRAPHS=0. We can run any of these, or a debug build, on request.

Environment

  • 2× Intel Arc Pro B70 32 GB (ASRock boards, subsystem 1849:6025), each in its own CPU root port, PCIe 5.0 x8
    each     (the x16 slot is bifurcated x8x8 through a C-Payne MCIO Gen5 retimer). Both cards are power-capped at
    230 W (power1_cap). Separate IOMMU groups, 32 GB ReBAR each.
  • AMD EPYC 4464P, 128 GB DDR5 ECC; ASRock Rack B650D4U-2L2T/BCM. Proxmox VE 9.2 (Debian 13), kernel
    7.0.14-20-pve, xe driver, GuC 70.72.1 / HuC 8.2.10 / DMC 2.6.
  • Engine image: Ubuntu 24.04 with Intel oneAPI 2026.1.1 (icpx 2026.1.1.20260724) and the compute runtime from
    the kobuk-team PPA: intel-opencl-icd / libze-intel-gpu1 / intel-ocloc 26.31.39395.14. Built with
    -DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_SYCL_PARITY=OFF.
  • Engine args (paths shortened):
    --pack packs/iq3_xxs --native …IQ3_XXS-00001-of-00002.gguf --ple-gguf …IQ3_XXS-00002-of-00002.gguf --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp mtp/rt --max-context 32768 --kv int8 --stream-experts --vram-reserve-mib 2048 --layer-split auto
    ONEAPI_DEVICE_SELECTOR=level_zero:gpu (from strata-sycl.sh), STRATA_VERIFY_DEVICE_PLAN=1.

Related

cc @maxfridbe. Thanks for the port: the single-card numbers on the B70 are excellent, and we'd love to run IQ3
across both cards.

Activity

  1. maxfridbe commented on Oct 6, 2026

    @maxfridbe
    maxfridbe
    on Oct 6, 2026

    on it

  2. maxfridbe
    added 2 commits that reference this issue on Oct 6, 2026

    sycl: the port refreshed to upstream 0.1.40 (main 1735d64); Niko1221#866

    8d6a46b

    sycl: the port refreshed to upstream 0.1.40 (main 82f46a8); Niko1221#866

    3f37281

  3. Niko1221 commented on Oct 7, 2026

    @Niko1221
    Niko1221
    on Oct 7, 2026
    Owner

    Thanks for the report. We have no Intel GPU to reproduce on. Please try, one at a time, --layer-split with an explicit first layer for the second card (for example --layer-split 20), then STRATA_WARM_GRAPHS=0, then STRATA_VERIFY_DEVICE_PLAN=0, and tell us which of them gets past the start-up hang. A debug build log from the hang would help; the planner's cache sizing on the first card only (the second card has 28 GiB free) looks like the place to start.

  4. crobe201 commented on Oct 7, 2026

    @crobe201
    crobe201
    on Oct 7, 2026
    Contributor

    Same setup here (2x Arc Pro B70, Flash-Next IQ3_XXS, --layer-split auto). On main at e8ca9af (v0.1.40.2), this still fails.
    #1111's fix (mirror only the first GPU's layers) is not in main's re-migration (23a38a0), and two further split regressions show up
    once it is patched. Details, patches and logs are in #1440. a5682fe (the 0.1.39 port) runs the split fine, 56–82 tok/s. Investigated with Claude Code.

  5. maxious
    added 3 commits that reference this issue on Oct 8, 2026

    sycl: subtract the DRM clients' total only when the driver reports th…

    704c38a

    benchy: the dual-config gate for the Niko1221#1054 sizing fix

    c7f2896

    docs: the benchy v1 baseline of upstream main on this box

    af94671
  6. Niko1221
    added a commit that references this issue on Oct 8, 2026

    SYCL layer split (#1440): the host mirror keeps to the first stage's …

    a853478

  7. djbrettb commented on Oct 8, 2026

    @djbrettb
    djbrettb
    on Oct 8, 2026
    Author

    Thanks, Niko. Here are results from the original 2× B70 setup for your checks, plus 0.1.41 and the STRATA_SH_STREAM=0 question from #1440. In short: 0.1.41 fixes the start-up on two B70s, and item 3 is still the blocker. The data below points at where to look.

    Setup

    • 2× Arc Pro B70, each PCIe Gen5 x8. EPYC 4464P, Proxmox VE 9 / kernel 7.0.14, xe.
    • Flash-Next IQ3_XXS, --layer-split auto, 32k context, int8 KV, --stream-experts, spec 4 + MTP.
    • ONEAPI_DEVICE_SELECTOR=level_zero:* set through the config's env.
    • AOT bmg-g31, clean builds with no local patches. Each run starts fresh.

    Your three checks, on 0.1.40.4 (6674a00)

    0.1.41 (fb58e0d)

    Run Start-up First request Decode
    default ✅ 25 s, 100% of the experts resident (26.5 + 24.3 GiB) ❌ the engine exits: Cannot submit to a queue with a dependency from a graph that is associated with a different context (verify.cpp:1706), and every retry repeats it –
    STRATA_SH_STREAM=0 ✅ same ✅ correct (408), 3 of 3 ~6 tok/s: 5.7 on short replies, 6.2 after a 2.5k prompt; 5-minute soak 12/12 OK, median 6.6

    That matches janneedgren's B60 result. The workaround avoids the crash but costs a lot of speed.

    The useful comparison: PR #1111 (3f37281)

    • On the same machine, model and config, it runs the split at 96 tok/s on short replies and 67 tok/s after a 2.5k prompt. It holds about 1,000 tok/s prompt read and 50–90 tok/s generation out to 202k context.
    • That port hard-wires sh_stream_on() to false.
    • So running without the side stream is not what makes main slow: Intel arc 0.1.40 #1111 runs without it at full speed, and main runs without it at ~6 tok/s.
    • Something else differs between Intel arc 0.1.40 #1111 and main's re-migration when the fork is off. Diffing the two around verify.cpp's window capture and replay is probably the quickest way to find it.
    • We have used 3f37281 as our daily coding model on 2× B70 since 2026-10-06, with 0 engine resets.

    Two small things

    • A --layer-split that finds only one device should refuse to start. Without the selector set, the image's level_zero:0 hid CUDA1, and the engine ran both stages on CUDA0 without saying so (layers 0-19 (CUDA0), 20-47 (CUDA0)).
    • strata-sycl.sh drops any value that is exactly 0, so STRATA_WARM_GRAPHS=0 and STRATA_SH_STREAM=0 never reach the engine. I passed 00.

    I'm happy to test a build with a graph fix on this pair. A run takes about 10 minutes.

  8. crobe201 commented on Oct 8, 2026

    @crobe201
    crobe201
    on Oct 8, 2026
    Contributor

    Second 2× Arc Pro B70 data point for #1111 (3f37281), clean build (AOT bmg-g31), only local change is --group-add 44 --group-add 993 in strata-sycl.sh (unprivileged Proxmox LXC).
    Box: EPYC 7F52, B70s at PCIe Gen4 x16 on different root complexes, Proxmox 9 / kernel 7.0.14, xe. Flash-Next IQ3_XXS, --layer-split auto (K=29, 100% resident), 32K ctx, int8 KV, spec 4 + MTP.

    Correct output (4/4 checks, greedy repeatable), no engine resets. Decode / prompt tok/s, single user:

    --ple-io direct (default) --ple-io ram
    short reply 38–59 64
    ~720-token prompt 71 / 277 95 / 303
    ~3.2K-token prompt 43 / 493 67 / 756

    Worth knowing for anyone comparing numbers: on a slow DRAM-less NVMe (our model files sit on one), the n-gram table's random 4 KB reads with the default --ple-io direct cost ~30–40% of decode speed. --ple-io ram (~29 GB of host RAM for IQ3_XXS) brings it in line with djbrettb's figures. Investigated with Claude Code.

  9. Niko1221 commented on Oct 13, 2026

    @Niko1221
    Niko1221
    on Oct 13, 2026
    Owner

    Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

    If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

  10. Niko1221
    closed this as not plannedon Oct 13, 2026
  11. giridhart
    added a commit that references this issue on Oct 15, 2026

    SYCL layer split (Niko1221#1440): the host mirror keeps to the first …

    a423df9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions