Repository navigation
Windows: the engine dies without a message (0xC0000409) when the commit limit is reached; the RAM checks only look at physical RAM #1607
Description
Summary
On Windows the engine exits without a message (exit code 3221226505 = 0xC0000409) when the system's commit charge
reaches the commit limit (RAM + page file), even with several GiB of physical RAM still available. The RAM admission
for the conversation cache and session checkpoints reads only ullAvailPhys (src/core/conversation_memory.cpp,
conversation_available_memory()), so it admits allocations the commit limit then refuses; expert_source.cpp already
reads ullAvailPageFile. Under WDDM the card's VRAM is charged to the commit as well (#141), so a page file setup
accepts (it warns below 4 GB, #60) can leave almost no headroom for the larger models.
Reproduced 6 times at two page-file sizes, with and without WSL; every crash came within 0.1 GiB of the limit and
right after a prompt read.
Setup
- RTX 5060 Ti 16 GB (driver 616.92), Ryzen 7 5700X3D, 80 GB RAM (79.9 GiB visible), Windows 11 Pro 10.0.26200
- Strata 0.1.40.3 (engine and server), release build
- unsloth UD-IQ4_XS (55.4 GiB of experts), config as setup writes it but without
--resident-budget-gib(see RTX 5000 PRO slow unsloth ud iq4_xs fix #1080):
--max-context 393216 --rope-scaling yarn --rope-scale 1.5 --kv int8 --kv-resident 32768 --expert-cache auto --spec 4 --pcie-frac 0.20 --spec-min-p 0.70;STRATA_TRACE=1 STRATA_RSS_TRACE=1inenv - strata.exe's private bytes: ~78 GiB right after loading, up to 79.2 GiB under load
- Load: OpenCode 1.18.35 in Docker in WSL 2 on three coding tasks (8-30K-token conversations with tool calls), or the
same 107 requests captured from that run and replayed from Windows with WSL shut down - Commit charge, commit limit, available RAM and strata.exe's private bytes sampled every 5-10 s
Results
| Run | Page file (commit limit) | Load | Extra commit | Commit max | Engine crashes |
|---|---|---|---|---|---|
| A | 16 GB (95.93 GiB) | agentic, WSL + Docker | - | 95.92 | 5 |
| B | 16 GB (95.93 GiB) | replay, WSL off | - | 92.0 | 0 |
| b8 | 32 GB (111.93 GiB) | agentic, WSL + Docker | 8 GiB | 102.48 | 0 |
| C | 32 GB (111.93 GiB) | replay, WSL off | until 0.8 GiB free | 111.86 | 1 |
"Extra commit" is a ballast process that commits memory with VirtualAlloc(MEM_COMMIT) and never touches it: the
commit rises, physical RAM does not (checked: +2.00 GiB commit, available RAM unchanged).
- A vs B: the same requests crash with WSL and Docker taking the last few GiB of commit, and not without them.
- b8: 30 minutes of the agentic load with the commit 3-6.5 GiB above the 16 GB limit and no crash; the limit, not
the load, is what fails. - C: the crash returns with the larger page file and without WSL once the commit is pushed to the limit.
- In one more run with the ballast at the limit (17.5 GiB, WSL + Docker), the WSL VM died instead
(Wsl/Service/0x80072746): whoever allocates next fails.
The samples just before each crash (available RAM, commit, strata.exe private bytes):
A 15:37:23 2.93 GiB 95.84 / 95.93 79.02 GiB -> 0xC0000409
A 15:41:30 3.21 GiB 95.89 / 95.93 79.05 GiB -> 0xC0000409
A 15:43:24 4.38 GiB 95.88 / 95.93 78.79 GiB -> 0xC0000005
A 15:47:00 4.01 GiB 95.88 / 95.93 79.15 GiB -> 0xC0000409
A 15:49:24 3.96 GiB 95.90 / 95.93 79.04 GiB -> 0xC0000409
C 01:14:14 7.60 GiB 111.86 / 111.93 78.93 GiB -> 0xC0000409 (fault offset 0x4eb1e9 in strata.exe)
Windows logged System event 2004 (Resource-Exhaustion-Detector, "virtual memory insufficient") ten times during A,
none during B or b8, and in C two minutes before the crash, for example:
Windows hat diagnostiziert, dass der virtuelle Speicher unzureichend ist. Die folgenden Programme belegten den meisten
virtuellen Speicher: strata.exe (21016) belegt 84826337280 Bytes, vmmemWSL (4584) belegt 4080967680 Bytes und
claude.exe (5908) belegt 623763456 Bytes.
The first occurrence, a day earlier with engine 0.1.40 in the same kind of agentic run, also had five crashes with
the same exit code and event 2004 at the same times.
Where it dies
Every crash came right after a prompt read, before the first decode step, which is where the checkpoints are saved.
The last engine log lines before each crash (STRATA_TRACE):
strata trace: read 1954 tokens (batched) in 7923.8 ms
strata trace: read 16 tokens (windows) in 377.0 ms
strata trace: read 833 tokens (batched) in 6420.5 ms
strata trace: read 65 tokens (batched) in 2379.8 ms
strata trace: read 3276 tokens (batched) in 8030.0 ms
strata verify: capturing the 6-token window (295 MiB of VRAM free)
The engine log has no error line; the server reports the engine stopped unexpectedly (exit code 3221226505).
0xC0000409 fits an allocation that throws where nothing catches it (std::terminate -> fail-fast). The conversation
cache's own catch (std::bad_alloc) in generate.cpp would have logged "allocation failed; skip parking", which does
not appear, so the failing allocation is elsewhere. Minidumps of the six crashes (~240 MB each) are available.
Suggestions
- Engine, Windows: let
conversation_available_memory()(and the session/checkpoint admission that uses it) return
min(ullAvailPhys, ullAvailPageFile), so a snapshot or checkpoint is skipped when the commit cannot take it. - Engine: catch
std::bad_allocaround the prompt-checkpoint saves as the conversation cache does, and install a
std::set_terminate/ unhandled-exception handler that writes one line ("out of memory: commit X of Y GiB") to the
log before exiting, so this case is recognisable. - Setup: the page-file warning (below 4 GB) does not catch this case. A check against the model's needs would: experts
in RAM + VRAM + pinned KV + a margin, compared with RAM + page file. Here: 55.4 GiB experts + 16 GiB VRAM + 4.6 GiB
KV = 76 GiB of 95.9 GiB before anything else runs. - Docs: "System managed" or a larger page file for the larger models on Windows. With 32 GB here (limit 111.9 GiB)
the same agentic runs had 18 GiB of headroom; the page file's peak use was 1.5 GB.
Activity
masel commentedon Oct 9, 2026
I have rewritten the description above with follow-up measurements, so it can be read on its own. What changed:
below the old limit. It is replaced by a run with extra commit (b8: 102.5 GiB, above the old limit, no crash) and a
reproduction with the 32 GB page file and without WSL (run C: crash at 111.86 of 111.93 GiB).
restarted after a crash).
The conclusion and the suggestions are unchanged.
Windows HIP: a bare --prefill auto stops at 6144; commit-limit check …
Niko1221 commentedon Oct 10, 2026
Thanks for the detailed measurements. Two changes in v0.1.42: the conversation cache and checkpoint admission on Windows now reads min(available RAM, available commit) instead of available RAM only, so a snapshot is skipped when the commit limit cannot take it; and serve start prints the commit left against the commit limit and warns when the planned host allocations (conversation cache budget plus checkpoints and staging) plus a 4 GiB floor do not fit, with the page-file advice. We have not added the catch-all handler that writes one line on a std::bad_alloc / terminate; that stays on the list. The silent 0xC0000409 may still come from an allocation we do not budget, so please send the next crash's engine log if it recurs.
masel commentedon Oct 12, 2026
Re-tested on v0.1.43 (engine 0.1.43), as asked: the silent exit still happens at the commit limit.
Same reproduction as run C in the description: UD-IQ4_XS without
--resident-budget-gib, 32 GB page file (limit 111.93 GiB), WSL shut down, the 107 captured agent requests replayed from Windows, and a ballast process that commits memory without touching it until 0.8 GiB of commit are left (it now ends as soon as the engine exits, so the restart is not starved).strata serve: Windows commit: 24.9 GiB left of a 111.9 GiB limit (RAM + page file)(before the ballast; no warning, as expected).STRATA_TRACE=1):skip parkingand noallocation failedline anywhere in this engine run, so neither guarded path was reached: it dies again right after a prompt read, where the prompt checkpoints are saved (checkpoint_at→conversation_checkpoint_save, which is not behind the admission). That matches my note on Windows: cap conversation-memory admission at the commit limit too (#1607) #1679.The full engine log and a minidump are available. With a terminate handler that logs one line, the failing allocation would show up directly.
Engine log excerpt
Niko1221 commentedon Oct 13, 2026
Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.
If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.
Windows HIP: a bare --prefill auto stops at 6144; commit-limit check …
masel commentedon Oct 16, 2026
Still happens on v0.1.44.1 — please reopen.
Same reproduction as run C in the description and as on 0.1.43 (UD-IQ4_XS without
--resident-budget-gib, 32 GB page file, WSL shut down, the 107 captured agent requests replayed from Windows, a ballast that commits untouched memory until 0.8 GiB of commit are left and ends with the engine):strata generate: session is up (engine 0.1.44.1)strata serve: Windows commit: 23.9 GiB left of a 111.9 GiB limit (RAM + page file)(before the ballast)0x5b8c29. Commit 111.83 of 111.93 GiB, 9.5 GiB of physical RAM available, strata.exe private 79.0 GiB.skip parking,allocation failedor any other memory line in the whole engine run:So the allocation that fails is still one the admission does not cover, most likely the prompt checkpoints taken after the read (
checkpoint_at→conversation_checkpoint_save). The full engine log and the minidump are available.