Skip to content

Peer-expert tier (--peer-device 1) crashes on first prompt: "iq_dequant_gu_f16: an illegal memory access" #2666

Closed as not planned

Description

@CaptRauti

Bug: two-GPU peer-expert tier (--peer-device 1) crashes on the first prompt — "iq_dequant_gu_f16: an illegal memory access"

Engine: 0.1.41 (locally built for this CPU)
OS: Windows 11 (WDDM)
Hardware: 2x NVIDIA RTX 3060 12 GB, Intel Xeon E5-2697 v2 (no AVX2), 128 GB RAM
Model: swift-1.5-iq2_xs (Qwen3.8-Flash-Next MoE), context 128000, KV q4_0, --kv-resident 32768, STRATA_ARENA_PIN_GIB=26

Reproduction

Run on two GPUs (config gpu: [0, 1]) with --peer-device 1, so CUDA1 becomes an adaptive expert-cache tier instead of a layer split. The engine starts cleanly and fills the peer tier, but the first ~5.3k-token prompt crashes it deterministically:

strata generate: --trim-stage-weights needs a layer split across GPUs (ignored)
strata generate: expert cache 3353 slots, 4.49 GiB of VRAM; policy is
strata generate: peer GPU 1: 7723 experts, 10.36 GiB (filled in 4.5 s); with the primary's 3353 that is 11076 of 24576 on the GPUs
strata generate: session is up (engine 0.1.41)
strata serve: prompt chunk auto: 7936 tokens, a 96-slot ring
strata prefill: peer GPU 1 computes its experts' rows of each prompt chunk (up to 79360 rows per layer, no P2P: through mapped host memory in FP16, one weighted sum per token back, compact group buffers); 0 MiB left free on it
strata serve: 443 MiB of VRAM free with everything loaded
iq_dequant_gu_f16: an illegal memory access was encountered

Crash site: src/prefill/prefill.cpp:3679 (iq_dequant_gu_f16 on the P6 native-pack dequant path).

Not a VRAM-shortage (already ruled out)

All three crash identically on the same first prompt:

  • --vram-reserve-mib 700 (baseline): 443 MiB free on CUDA0, peer card 0 MiB free
  • --vram-reserve-mib 1800 (primary reserve raised): 1497 MiB free on CUDA0 — same crash
  • --prefill auto:8192 (smaller prompt chunks, per the engine's own startup hint) + --peer-reserve-mib 1200: same crash, peer card 679 MiB free

With no peer tier the same model+prompts run fine on the same two cards as a layer split.

Why this matters

The peer tier is a big win for small cards: primary 3353 + peer 7723 = 11076 of 24576 experts on the GPUs, versus ~5,700 on the two-card layer split for the same model. If the peer prefill path could run without the illegal access, small-VRAM dual-GPU boxes would get a substantial decode lift.

Repro args: --serve --pack ... --expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.5 --mtp ... --max-context 128000 --kv q4_0 --kv-resident 32768 --peer-device 1

Happy to attach the full engine logs or run further diagnostics.

Activity

  1. added a commit that references this issue on Oct 17, 2026
  2. Niko1221 commented on Oct 19, 2026

    @Niko1221
    Owner

    Thanks for taking the time to report this.

    We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

    To reopen it takes about two minutes:

    1. Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
    2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
    3. Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
    4. Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

    The rules are in CONTRIBUTING.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions