Skip to content

[Engine] checkpoint save: "conversation snapshot running-state copy: out of memory" exits the engine (21 times in 4 h), while parking admission refuses the same conversation (0.1.45, 2x RTX 3080 20 GB, 62.7 GiB RAM, ~310K contexts) #3350

New issue
New issue
Closed
Closed
[Engine] checkpoint save: "conversation snapshot running-state copy: out of memory" exits the engine (21 times in 4 h), while parking admission refuses the same conversation (0.1.45, 2x RTX 3080 20 GB, 62.7 GiB RAM, ~310K contexts)#3350

Description

@WarlockSyno
WarlockSyno
opened on Oct 20, 2026

Area

[Engine] (speed, kernels, models, the server and API)

Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

What happened

While serving two alternating long conversations (~310K tokens each) on a two-gpu layer split, the engine exited 21 times in 4 h 08 min (exit code 1, 2026-10-19 19:20:21 .. 23:02:57), each time right after a conversation checkpoint save ran out of host RAM. Every exit was followed by the runtime reloading the model (~45 s) and the next request succeeding, so the service looked healthy throughout.

The engine log shows the same three lines each time:

strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed

and the runtime reports:

[strata] the engine stopped unexpectedly (exit code 1). The engine exited (code 1) after it reported: saving a checkpoint part failed - please report it at github.com/Niko1221/Strata/issues with the log. The next request starts the engine again.
[strata] done: 0 tokens in 2 s (0.0 tok/s) (error, cancel=False)
[strata] the engine had stopped (exit code 1); starting it again (a minute or two) ...

The part I did not expect: the parking admission gate refuses the very same conversation moments earlier, on the same host, and it is right.

strata serve: conversation cache: skip parking (physical RAM admission; need 2915 MiB plus 1536 MiB floor, 2437 MiB available, source meminfo; evicted 0, 0 still parked)
strata serve: conversation cache: parked 310875 tokens in 190.5 ms; parked=1 bytes=6022875424 evictions=0 snapshot_bytes=6022875424 reused_kv_bytes=4740933120

So parking declines cleanly when the RAM is not there, and then the checkpoint writer copies anyway and takes the engine down with it. Numbers on this box at the time of the failure: engine RSS 51.7-54.8 GiB of 62.7 GiB RAM, one parked snapshot of a 310K-token conversation 5.61 GiB (6022875424 bytes). With --conversation-cache-slots 3 that is a planned ~17 GiB of snapshots on the few GiB that are left.

What I expected instead: a conversation checkpoint is an optimization, the same way parking is. If the copy cannot be made, drop that checkpoint (or that conversation's entry) and serve the request, rather than exiting the engine. As it stands a refused optimization becomes a 45 s outage, every parked conversation is lost, and in-flight requests fail.

Client-visible damage in the same window: done: 0 tokens in 2 s (0.0 tok/s) (error, cancel=False), and one request died after 1519 generated tokens.

Also worth knowing for anyone monitoring this: systemd never sees it. The reported unit stayed active, NRestarts=0, Result=success for the whole 4 h 08 min run, because the engine is a child of serve/server.py and the restart happens inside the runtime. The only reliable signal is journalctl -b -u strata | grep -c "engine stopped unexpectedly".

Smallest steps or command that reproduce it

  1. Two-gpu layer split with long contexts, conversation caching on with more slots than the RAM can hold:
--layer-split 25 --max-context 524288 --kv int8 --spec 4 \
--conversation-cache-mib 12288 --conversation-cache-slots 3 \
--conversation-cache-min-free-mib 1536 --session-save-reclaim

  1. Serve two conversations of ~300-310K tokens alternately from two clients (mine came from two connections on the same LAN host). In my log the last three failures arrived exactly 6:00 apart.

  2. The failures stop when the slot count fits the RAM. With --conversation-cache-slots 1 and nothing else changed, the same two clients, the same ~310K requests: 13 requests over 13.5 min, 0 engine exits, 0 out-of-memory lines (largest prompt 311,068 tokens). Service memory peak 58 GiB with 1.9 GiB swap, against 59.2 GiB + 6.8 GiB swap before.

There is one other exit in the same boot with the same wording, from a different path: verify: instantiate: out of memory also exits the engine (1 occurrence). Same shape, same expectation.

Reproduction status

Reproduced locally

Strata version

Source install, v0.1.45, commit 477ac13c, engine/BUILD.json "version": "0.1.45" (GET /props reports Strata 0.1.45).

Graphics card, memory and system

2 x RTX 3080 20 GB (sm_86), 62.7 GiB RAM (MemTotal: 65728088 kB), CachyOS, kernel 7.2.8-1-cachyos, NVIDIA driver 615.71.09, CUDA 13. zram for swap (62.7 GB, 1.2 G in use), since that matters below.

Model, size and settings

Qwen3.8-Flash-Next IQ3_S (native pack), --max-context 524288 (--rope-scaling yarn --rope-scale 2.0, trained 262144), --kv int8, --spec 4, --mtp, --vision, --expert-cache auto, --layer-split 25, --vram-reserve-mib 512, --pcie-frac 0.20, --conversation-cache-mib 12288, --conversation-cache-slots 3, --conversation-cache-min-free-mib 1536, --session-save-reclaim.

Logs

The failure, with the admission refusal that preceded it (engine log, one occurrence):

strata serve: conversation cache: skip parking (physical RAM admission; need 2715 MiB plus 1536 MiB floor, 3442 MiB available, source meminfo; evicted 0, 0 still parked)
strata serve: conversation cache: restored 306479 tokens (live) in 836.2 ms; parked=0 bytes=5029207144
strata serve: prompt 306705 tokens = 306479 reused + 226 read in 2119 ms (106.7 tok/s), 348 generated in 4309 ms (80.8 tok/s), drafts accepted 184 of 236, 6 checkpoints
strata serve: conversation cache: parked 307054 tokens in 220.0 ms; parked=1 bytes=5872999448 evictions=0 snapshot_bytes=5872999448 reused_kv_bytes=4678049664
strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed

Exit codes for the whole boot: 21 x (exit code 1) on this path, 1 x (exit code -15) which was my own systemctl restart.

Snapshot sizes observed for 309-310K-token conversations: 5.61-6.02 GB (bytes=6022875424, bytes=5872999448).

Checked against 0.2.0-beta / engine 0.1.45.3 (static analysis, not run)

I did not run 0.2.0-beta on this box, so this part is from the source, not from a measurement:

  • src/core/conversation_state.cpp is byte-identical between the v0.1.45 tree and v0.2.0-beta (diff is empty), so the copy that fails is the same code.
  • The call site in src/program/generate.cpp is unchanged in behaviour: if (!checkpoint_save(part, stage_ss, g)) { e = "saving a checkpoint part failed"; return false; }, still propagating to the caller's if (!saved) { e = "saving a conversation checkpoint failed" + ckpt_why; return false; }.
  • Nothing in the 0.2.0 release notes mentions conversation-cache checkpoint memory.

If a memory admission check for this path did land somewhere I did not look, I would rather be corrected than file a duplicate, and I will happily re-test on a 0.2.0-beta build.

One thing that may be relevant to how you want to fix it

The RAM guard already computes exactly what the parking path needs (need N MiB plus M MiB floor, K MiB available, source meminfo), and it consults MemAvailable plus cgroup limits. The checkpoint writer appears to consult nothing. zram is not a safety net here: it is compressed RAM, so the spike has nowhere to go, and the copy is a cudaMemcpy to host memory that returns CUDA_ERROR_OUT_OF_MEMORY rather than an allocator failure the caller could catch earlier.

Activity

  1. github-actions commented on Oct 20, 2026

    @github-actions
    github-actionsBot
    on Oct 20, 2026 – with GitHub Actions

    Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:

    • Fill in "Area".
    • Fill in "Desktop app, Terminal or source?".

    Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment /reopen.

    Details: CONTRIBUTING.md

  2. github-actions
    added
    needs-formatTitle or template does not follow CONTRIBUTING.md yet
    on Oct 20, 2026

  3. github-actions commented on Oct 20, 2026

    @github-actions
    github-actionsBot
    on Oct 20, 2026 – with GitHub Actions

    Thanks, this follows the format now.

  4. github-actions
    removed
    needs-formatTitle or template does not follow CONTRIBUTING.md yet
    on Oct 20, 2026

  5. sanastasiou commented on Oct 25, 2026

    @sanastasiou
    sanastasiou
    on Oct 25, 2026
    Contributor

    Same crash here, on a 2× RTX 3090 layer split. Adding data points, one of which may matter for #4073's theory.

    Setup

    • Strata 0.2.2.
    • Qwen3.8-Flash-Next IQ3_S. Two engines on one host, each a 2× RTX 3090 layer split (PCIe 3.0, no NVLink), sharing 251 GiB of host RAM.
    • --max-context 524288 --batch-groups auto --pipeline-windows 2 --prompt-cache-every 2048 --trim-stage-weights
    • Conversation cache: 24 GiB, 8 slots, --conversation-cache-min-free-mib 40960, 48 prompt checkpoints.
    • Traffic: coding agents, long sessions (60K–360K-token prompts), alternating conversations.

    Frequency

    • 16 exits in 34 h on 0.2.2: 10 on one engine, 6 on the other.
    • Before that: 16 on 0.1.43 and 3 on 0.2.2 during 22–23 Oct.

    Always the same two lines, right after a conversation was parked:

    strata serve: conversation cache: parked 71782 tokens in 1642.8 ms; parked=7 bytes=21706098276 evictions=159 snapshot_bytes=4105009932 reused_kv_bytes=1063041792
strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed

    The runtime then logs the engine stopped unexpectedly (exit code 1) ... The next request starts the engine again.

    Observations

    1. Raising the RAM floor didn't help. Going from 20480 to 40960 MiB did not stop it. The floor only gates parking admission, not the checkpoint copy. At the time of a check after the latest crash, ~130 GiB of host RAM was still available.
    2. No graph-instantiation OOM recoveries came first. Our engine log contains no graph-instantiation OOM recovery lines at all. So in our case the copy fails without the graph-cache growth [Engine] Ordinary batch: checkpoint-copy OOM after graph-cache growth in v0.2.3.1 #4073 describes. The two cards of the crashing pair held ~23.6–23.9 of 24 GiB just before, so VRAM headroom was thin.
    3. Lazy restart plus a load balancer = permanently dead leaf. Because the restart waits for "the next request", an engine behind a balancer that marks it unhealthy never gets that request. /health keeps answering {"status": "ok", "loaded": false}. Our leaf stayed down for ~1.5 h until we noticed. Suggestions:
      • restart on exit regardless of traffic (or make that an option);
      • make status reflect a dead engine.

    What would fix it for us: a failed checkpoint save should drop that checkpoint (and maybe free the oldest parked conversation), not exit the engine. Even better, reserve the staging buffer for the running-state copy up front, so the copy can't hit out-of-memory mid-request.

    Related: #4073 (same model, 2× 3090, 0.2.3.1).

  6. KodeMunkie commented on Oct 26, 2026

    @KodeMunkie
    KodeMunkie
    on Oct 26, 2026

    Same issue here on a Docker deployment serving an agent, with a two-GPU layer split. Adding another hardware/configuration data point.

    Environment

    • Strata CUDA nightly built 25 October 2026, source commit 4b9e05a2a806.
    • 2 × NVIDIA RTX 5060 Ti, 16 GB VRAM each.
    • 157 GiB system RAM, 23 GiB swap.
    • Linux 7.0.0-38-generic, NVIDIA driver 595.99.02.
    • Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF, UD-IQ4_XS.
    • Vision enabled.

    Relevant effective engine settings:

    --max-context 262144
--kv k8v4
--kv-resident 32768
--vram-reserve-mib 512
--expert-cache auto
--prefill auto
--layer-split auto
--batch 2
--spec 3
--spec-min-p 0.05
--ple-io ram
--conversation-cache-mib 16384
--conversation-cache-slots 4

    Observed failure

    The retained engine log contains three occurrences with this model and the 512 MiB reserve configuration. There are ten occurrences overall across earlier models/configurations too.

    Two examples from the current configuration:

    strata serve: conversation cache: parked 58692 tokens in 274.0 ms; parked=4 bytes=4108387376 evictions=20 snapshot_bytes=845753880 reused_kv_bytes=0
strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed

    strata serve: conversation cache: parked 104644 tokens in 1239.9 ms; parked=4 bytes=7045935728 evictions=1 snapshot_bytes=2128500792 reused_kv_bytes=0
strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed

    The server then reports that the engine exited with code 1 and that the next request starts it again. Affected requests report zero output tokens.

    Memory observations

    Startup logs for an affected run reported only 205 MiB free on the first GPU and 77 MiB on the second, before subsequent graph captures.

    Host-side checks during the investigation found approximately 6–154 MiB free on the two serving GPUs. System RAM availability was approximately 37–43 GiB, with almost no swap used. These are investigation-time samples, not measurements captured at the exact failing CUDA call.

    This supports investigating GPU/driver resource pressure, but does not establish which allocation failed or why. I cannot distinguish stream creation, copy staging or another allocation from the existing error message.

  7. Niko1221
    added 5 commits that reference this issue on Oct 27, 2026

    serve: a checkpoint whose copy runs out of memory is skipped, it no l…

    f62589f

    conversation copies: the private copy streams and events are handed b…

    3478382

    serve: an idle batch slot's decoded conversation is parked before an …

    01e404c

    Later-card VRAM: audit fixes for failed allocations, the reserve arit…

    2a9a67c

    Conversation cache: skip a restore only when a card would have to cre…

    8c03014

  8. Niko1221 commented on Oct 27, 2026

    @Niko1221
    Niko1221
    on Oct 27, 2026
    Owner

    Thank you. Fixed in 0.2.7: A checkpoint whose copy runs out of memory is skipped, it no longer ends the engine.

    Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7

  9. Niko1221
    closed this as completedon Oct 27, 2026
  10. DamirLukina88
    added a commit that references this issue on Oct 27, 2026

    fix(engine,serve): port four upstream v0.2.7 fixes

    181f77e
  11. revaraver
    added 3 commits that reference this issue on Oct 27, 2026

    serve: a checkpoint whose copy runs out of memory is skipped, it no l…

    16a8339

    conversation copies: the private copy streams and events are handed b…

    9b58fd6

    Conversation cache: skip a restore only when a card would have to cre…

    ea4ff99
  12. lask3802
    added a commit that references this issue on Oct 27, 2026

    remote stage: a failed checkpoint save still ends the request

    8874f76
  13. riceharvest
    added a commit that references this issue on Oct 30, 2026

    sycl: re-migrate the port to upstream 0.1.54 (from its last re-migrat…

    8adf90c
  14. MistVVK
    added a commit that references this issue on Oct 30, 2026

    serve: チェックポイントの写しがメモリ不足のときは飛ばし、エンジンを止めない（upstream f62589f）

    5f2c261
  15. ruc1j
    added a commit that references this issue on Oct 31, 2026

    engine: Niko1221#3416 の組、AVX-512 の gather の行、会話の退避とグラフの解放を移植する（Issue #51

    b3aa4c4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions