Skip to content

Every engine exit is reported as "probably out of RAM", even when the engine's log names the cause #215

Closed

Description

@alphastorm

When the engine process ends during a request, the server prints death_note() (serve/server.py#L205-L221, used at #L856-L862). Unless the end of the log has the watchdog's issue #29 line, the note is always "The usual cause is running out of RAM ... Close other programs or use a smaller model". It does not look at the exit code or at the engine's own last log line.

We saw this on Windows 11 after the crash from #183 (exit code 1, the log ends with strata serve: cancelled) and after deliberately killing the engine mid-generation; neither had anything to do with memory. The note points users at RAM while the log already names the cause, and by its own text it cannot apply on Windows ("Windows slows down instead").

Reproduce (no GPU)

A stand-in engine that fails its first request the way #183's engine does, saved as fake_engine.py (executable) in the checkout root:

#!/usr/bin/env python3
import sys
print("READY 32768 stop", flush=True)
for line in sys.stdin:
    if line.startswith("GEN"):
        sys.stderr.write("strata serve: cancelled\n"); sys.stderr.flush()
        print("ERR cancelled", flush=True)
        sys.exit(1)
import sys, threading
sys.path.insert(0, "serve")
from server import StrataEngine

open("fake_engine.log", "w").close()
eng = StrataEngine("./fake_engine.py", [], log="fake_engine.log")
for attempt in (1, 2):
    try:
        for _ in eng.generate([1, 2, 3], 8, {}, threading.Event()):
            pass
    except Exception as e:
        print(f"request {attempt}: {type(e).__name__}: {e}")
print("log tail:", open("fake_engine.log").read().strip().splitlines()[-1])
print("death_note:", eng.death_note())

Output:

[strata] starting the engine: reading the model's weights ...
request 1: ValueError: cancelled
request 2: EngineDied: the engine stopped unexpectedly (exit code 1)
log tail: strata serve: cancelled
death_note: The usual cause is running out of RAM: Linux then ends the biggest program (check: sudo dmesg | grep -i -E 'killed process|out of memory'); Windows slows down instead. Close other programs or use a smaller model (Q2_0 / IQ2_XS).

Suggested

Give the RAM explanation only when the exit looks like the OS killed the engine (on Linux a negative return code, -9 for the OOM killer's SIGKILL), and otherwise print the exit code and the last strata ... line of the log, which death_note() already reads.

Activity

  1. Niko1221 commented on Sep 30, 2026

    @Niko1221
    Owner

    Agreed. In 0.1.28 (in testing) the message quotes the engine's exit code and its last log line; the RAM hint only appears when the OS killed the engine.

  2. Niko1221 commented on Sep 30, 2026

    @Niko1221
    Owner

    Fixed in 0.1.28: the message quotes the engine's exit code and last log line; the RAM hint only appears when the OS killed the engine.

  3. added a commit that references this issue on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions