Heap corruption (munmap_chunk: invalid pointer) at startup in Prefill::init — crashes with 1/2/4 GPUs #828
Description
Crash
Engine crashes at startup with munmap_chunk(): invalid pointer (heap corruption) during prompt path initialization. Happens consistently, 100% reproducible.
Log (last lines before crash)
strata generate: layer split: CUDA0 runs layers 0-11
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 8 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.39)
strata generate: token graph hit path: 16530 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
munmap_chunk(): invalid pointer
With default settings (no STRATA_SPLIT_OWN), it crashes at:
strata serve: CUDA3 prompt path borrows 2914 of its 4526 slots (4.03 GiB)
munmap_chunk(): invalid pointer
System
- 4x RTX 3060 12GB (sm_86)
- 128GB DDR4-3200
- Debian 13 (trixie), glibc 2.41
- CUDA 13.0 toolkit, NVIDIA driver 595.91.07
- Engine 0.1.39 (commit 6f32ec0, latest main as of 2026-10-04)
- Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS, 262144 context, --vision yes
- Running in LXC container
What I've tried
- 1, 2, and 4 GPUs (--gpus 0 / 0,1 / all) — all crash identically
- With and without STRATA_SPLIT_OWN=1 — both crash
- STRATA_RING_BYTES=0 (opt out of Prefill chunk ring #583) — still crashes
- 32k vs 262k context — both crash
- Increased --vram-reserve-mib to 2000 — still crashes
- MALLOC_CHECK_=3 confirms heap corruption (process aborts instead of munmap error)
Analysis
The crash is inside Prefill::init() (src/prefill/prefill.cpp). The borrow buffer (from ExpertCache::device_slot()) is valid — I verified the slot index is in-bounds and the pointer is non-null. The corruption happens after Prefill::init receives the borrow buffer.
The #583 "byte-budget ring" changes are the likely culprit (all recent changes to prefill.cpp are from #583), but STRATA_RING_BYTES=0 does not fully disable the code path.
This is different from Issue #70 (which crashes mid-decode after successfully serving). This crashes at startup, before serving any requests.
Activity
wehooper4 commentedon Oct 4, 2026
Note: this is NOT the same as #133.
#133 is VRAM exhaustion on the GPU (
fatal: Memory allocation failure/cublasCreate failed) — the GPU runs out of memory during prompt prefill init.#828 (this issue) is host heap corruption (
munmap_chunk(): invalid pointer) — glibc detects corrupted heap metadata on the CPU side. The GPU memory is fine; it's the host-sidePrefill::init()that's corrupting the heap.Different failure modes, different root causes. The "prompt path borrows/allocates" log line appears in both, but that's just the last thing printed before two different crashes.
prefill: STRATA_TEST_PAGEABLE=1 forces the pageable fallbacks of Pref…
Niko1221 commentedon Oct 6, 2026
We could not find this one by reading. 0.1.40 adds STRATA_TEST_PAGEABLE=1, which forces the pageable fallbacks of the prompt path on a PC that can pin memory (it ran clean on Windows here, but Windows has no heap checker). Could you run with MALLOC_CHECK_=3 under gdb (bt), send 'ulimit -l' from inside the container, and try once outside LXC? Pinned allocations failing under the container's memlock limit is our main suspect.
prefill: STRATA_TEST_PAGEABLE=1 forces the pageable fallbacks of Pref…
prefill: STRATA_TEST_PAGEABLE=1 forces the pageable fallbacks of Pref…
Niko1221 commentedon Oct 7, 2026
Please run the engine with MALLOC_CHECK_=3 under gdb and send the backtrace of the munmap_chunk invalid pointer, with ulimit -l and the output of nvidia-smi. A run outside LXC would also help us tell the container apart. 0.1.40 has an STRATA_TEST_PAGEABLE=1 test switch if you want to compare.
prefill: STRATA_TEST_PAGEABLE=1 forces the pageable fallbacks of Pref…
Niko1221 commentedon Oct 13, 2026
Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.
If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.