Repository navigation
Multi-GPU layer split aborts on CUDA1 with fatal : Memory allocation failure / prefill gemm: cublasCreate failed #133
Description
Strata Version: 0.1.22 prebuilt (Windows x64, CUDA 13.0)
Hardware:
- GPU 0: NVIDIA GeForce RTX 5060 Ti 16 GB (PCIe x4, ~6.5 GB/s)
- GPU 1: NVIDIA GeForce RTX 5070 Ti 16 GB (PCIe x16, ~28.4 GB/s)
- CPU: AMD Ryzen 7 5800X (AVX2)
- RAM: 64 GB
Model / Config:
- Qwen3.8-Flash-Next IQ3_S
- Setup options: Dual GPU (
0, 1), 64K context, int8 KV, vision disabled.
Description:
Running START-HERE.bat with dual-GPU enabled completes model loading, expert arena pinning, and fills the expert caches (CUDA0: 4096 slots / 7.50 GiB; CUDA1: 3078 slots / 5.93 GiB).
Immediately after the token graph hit path is initialized, strata serve crashes when attempting to initialize the prompt prefill path:
strata generate: layer split: CUDA1 runs layers 8-47, expert cache 3078 slots (5.93 GiB), 3078 of its 20480 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-7
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 7174 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
fatal : Memory allocation failure
fatal : Memory allocation failure
fatal : Memory allocation failure
strata serve: layer split, CUDA1 prompt path: prefill gemm: cublasCreate failed
last 100 lines of log
PS D:\AI\lm_models\Strata> Get-Content -Tail 100 .\strata-iq3_s.log
strata-vision: warmed up at 1024 image tokens
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 20)
strata generate: PCIe probe: 6.5 GB/s host->device -> pcie_frac 0.14 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: D:\AI\lm_models\Strata-data\packs\iq3_s experts (largest blob 2.66 MB), token embedding type 23 in mapped host memory (322 MiB)
strata generate: 1466 MiB of weights loaded from D:\AI\lm_models\Strata-data\packs\iq3_s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights
strata generate: PLE on, table 320001536 rows of D:\AI\lm_models\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 PCIe probe 27.9 GB/s -> pcie_frac 0.55
strata generate: layer split: CUDA1 holds its weights, session and head; 9.53 GiB free
strata mtp: draft layer loaded, 876 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: locked 40290 MiB via working-set minimum + VirtualLock; cudaHostRegister limited to 8 GiB for CUDA1; 8 slices pinned (7 GiB); large pages refused for 50295996416 B (GetLargePageMinimum=2097152, VirtualAlloc error 1450); using 4 KB pages
strata generate: loaded 46.84 GiB at 2.32 GiB/s
strata generate: profile D:\AI\lm_models\Strata\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: expert cache auto: 10.16 GiB free, 2048 MiB reserved -> 3094 slots
strata generate: only 1614 MiB free once the slots are written (reserve 2048 MiB); shrinking the expert cache
strata generate: expert cache 4002 slots, 7.18 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 4002 of 4002 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 20-47, expert cache 2615 slots (5.18 GiB), 2615 of its 14336 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-19
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 6617 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata serve: layer split, CUDA1 prompt path: prefill gemm: cublasCreate failed
strata generate: --expert-cache-remote with a layer split needs a GPU that runs no stage (2 visible, 2 used by the split)
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: PCIe probe: 6.5 GB/s host->device -> pcie_frac 0.14 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: D:\AI\lm_models\Strata-data\packs\iq3_s experts (largest blob 2.66 MB), token embedding type 23 in mapped host memory (322 MiB)
strata generate: 1466 MiB of weights loaded from D:\AI\lm_models\Strata-data\packs\iq3_s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights
strata generate: KV streaming: 32768 of 65536 cells per QSA layer in VRAM, the K/V in 0.77 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of D:\AI\lm_models\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 PCIe probe 28.4 GB/s -> pcie_frac 0.55
strata generate: layer split: CUDA1 holds its weights, session and head; 9.92 GiB free
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: locked 40290 MiB via working-set minimum + VirtualLock; cudaHostRegister limited to 8 GiB for CUDA1; 8 slices pinned (7 GiB); large pages refused for 50295996416 B (GetLargePageMinimum=2097152, VirtualAlloc error 1450); using 4 KB pages
strata generate: loaded 46.84 GiB at 2.32 GiB/s
strata generate: profile D:\AI\lm_models\Strata\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.57 GHz -> 0.78 ms per layer, 8.38 GiB for experts
strata generate: layer split auto: CUDA1 70 SMs at 2.48 GHz -> 0.42 ms per layer, 5.93 GiB for experts
strata generate: layer split auto: K=8 - predicted 32.7 ms per decode window; the caches hold 7174 of 24576 profiled pairs (~94.9% of the routed mass)
strata generate: expert cache auto: 10.55 GiB free, 700 MiB reserved -> 3379 slots
strata generate: expert cache 4096 slots, 7.50 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 4096 of 4096 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 8-47, expert cache 3078 slots (5.93 GiB), 3078 of its 20480 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-7
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 7174 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
fatal : Memory allocation failure
fatal : Memory allocation failure
fatal : Memory allocation failure
strata serve: layer split, CUDA1 prompt path: prefill gemm: cublasCreate failed
strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)
strata generate: PCIe probe: 6.5 GB/s host->device -> pcie_frac 0.14 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: D:\AI\lm_models\Strata-data\packs\iq3_s experts (largest blob 2.66 MB), token embedding type 23 in mapped host memory (322 MiB)
strata generate: 1466 MiB of weights loaded from D:\AI\lm_models\Strata-data\packs\iq3_s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights
strata generate: KV streaming: 32768 of 65536 cells per QSA layer in VRAM, the K/V in 0.77 GiB of pinned RAM
strata generate: PLE on, table 320001536 rows of D:\AI\lm_models\Strata-data\models\IQ3_S\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
strata generate: layer split: CUDA1 PCIe probe 28.3 GB/s -> pcie_frac 0.55
strata generate: layer split: CUDA1 holds its weights, session and head; 9.92 GiB free
strata mtp: draft layer loaded, 835 MiB of VRAM (experts 675, dense 111)
strata generate: expert arena: locked 40290 MiB via working-set minimum + VirtualLock; cudaHostRegister limited to 8 GiB for CUDA1; 8 slices pinned (7 GiB); large pages refused for 50295996416 B (GetLargePageMinimum=2097152, VirtualAlloc error 1450); using 4 KB pages
strata generate: loaded 46.84 GiB at 2.35 GiB/s
strata generate: profile D:\AI\lm_models\Strata\data\expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: layer split auto: CUDA0 36 SMs at 2.57 GHz -> 0.78 ms per layer, 8.38 GiB for experts
strata generate: layer split auto: CUDA1 70 SMs at 2.48 GHz -> 0.42 ms per layer, 5.93 GiB for experts
strata generate: layer split auto: K=8 - predicted 32.7 ms per decode window; the caches hold 7174 of 24576 profiled pairs (~94.9% of the routed mass)
strata generate: expert cache auto: 10.55 GiB free, 700 MiB reserved -> 3379 slots
strata generate: expert cache 4096 slots, 7.50 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 4096 of 4096 slots from the profile; slot 0 verified
strata generate: layer split: CUDA1 runs layers 8-47, expert cache 3078 slots (5.93 GiB), 3078 of its 20480 profiled pairs; slot 0 verified
strata generate: layer split: CUDA0 runs layers 0-7
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 7 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.22)
strata generate: token graph hit path: 7174 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
fatal : Memory allocation failure
strata serve: layer split, CUDA1 prompt path: prefill gemm: cublasCreate failed
PS D:\AI\lm_models\Strata>
Workarounds Attempted (All Failed)
I attempted numerous manual parameter adjustments and clean installs to isolate the cause:
Lowering Context Capacity:
- Reduced --max-context from default 262144 down to 131072, and finally to 65536.
- Result: KV streaming freed up some VRAM (0.77 GiB of pinned RAM), but the engine simply expanded the expert cache slots to occupy the freed space, leading to the exact same crash at the same point.
Disabling Vision:
- Turned --vision off completely and wiped the vision sub-object from the run JSON to rule out strata-vision.exe VRAM reservations (~1.2 GB).
- Result: Same cublasCreate failed on CUDA1.
Lowering Prefill Chunk Size:
- Dropped --prefill from auto down to 2048, and then to 512 to shrink the prompt GEMM activation workspace.
- Result: When prefill was 2048, it threw prefill: device buffers for a chunk of 2048 tokens do not fit. When lowered to 512, it threw fatal : Memory allocation failure during cublasCreate.
Manual Layer Split Balancing:
- layer_split auto assigned layers 0–7 to CUDA0 (5060 Ti) and layers 8–47 to CUDA1 (5070 Ti).
- Manually tested --layer-split 20 to force layers 0–19 onto CUDA0 and 20–47 onto CUDA1.
- Result: The engine still sized CUDA1's expert cache aggressively up to the remaining VRAM ceiling, failing immediately at cublasCreate.
Increasing VRAM Reserve Floor:
- Added --vram-reserve-mib 2048 (up from default 700).
- Result: The reserve was respected on CUDA0 only (only 1614 MiB free once the slots are written (reserve 2048 MiB); shrinking the expert cache). CUDA1 completely ignored the reserve and allocated down to the edge.
Clean Reset:
- Completely deleted strata-iq3_s.json and re-ran START-HERE.bat --setup from scratch choosing recommended 64K context and no vision.
- Result: Still failed during calibration / first start on CUDA1 with fatal : Memory allocation failure.
Root Cause Analysis
In multi-GPU pipeline splits, strata.exe appears to suffer from an allocation sequencing problem combined with an unconstrained auto-cache calculation on secondary devices:
- Unreserved Secondary Allocation: The auto-cache allocator sizes and populates the expert cache on CUDA1 based on total raw free VRAM without factoring in the --vram-reserve-mib floor or leaving room for cuBLAS runtime handles.
- Buffer Borrowing vs. Standalone Buffers: On single-GPU, prompt prefill borrows slots from the expert cache. In multi-GPU split mode, it prints:
- the prompt path allocates its own buffers (too few cache slots to borrow)
- Because dedicated buffers are created after the expert cache has already consumed all remaining VRAM, cublasCreate() lacks the ~300–800 MiB contiguous workspace it needs to initialize on CUDA1.
Suggested Fixes
- Enforce the --vram-reserve-mib buffer calculation globally across all devices in multi mode, not just CUDA0.
- Initialize cublasCreate() and allocate the prompt prefill workspace buffers before calculating the remaining free VRAM available for the expert cache.
(Used AI to help me write this)
Activity
Niko1221 commentedon Sep 29, 2026
Thanks for the detailed report. The reserve is actually applied on CUDA1 as well.
fatal : Memory allocation failurepoints to Windows' commit limit rather than VRAM: every allocation on each card is also charged to RAM + page file (#60), and two full 16 GB cards plus 47 GB of experts is more than 64 GB. Could you set the page file to System managed (or at least 32 GB), try 0.1.26, and tell us the Commit figure from Task Manager if it still fails? We'll also make the second GPU keep room for its own cuBLAS start-up, and have setup check the commit limit for multi-GPU.Niko1221 commentedon Sep 30, 2026
Closing for now since we haven't heard back. If it still fails on 0.1.28, please reopen or open a new issue with the page-file setting, the Commit figure from Task Manager while it starts, and the end of the log.
tools/strata_mcp.py: an MCP server so an AI assistant can install, st…
Engine 0.1.34: AMD on Windows (a ready-made HIP engine, #247 #325), a…
tools/strata_mcp.py: an MCP server so an AI assistant can install, st…
Engine 0.1.34: AMD on Windows (a ready-made HIP engine, #247 #325), a…
tools/strata_mcp.py: an MCP server so an AI assistant can install, st…
Engine 0.1.34: AMD on Windows (a ready-made HIP engine, Niko1221#247 N…
tools/strata_mcp.py: an MCP server so an AI assistant can install, st…
Engine 0.1.34: AMD on Windows (a ready-made HIP engine, Niko1221#247 N…