[Bug]: Nightly 1400/1414 backend OOMs on startup replaying full event log for projection.attachment-cleanup

[alecramos-sudo]

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Run any nightly before 0.0.41-nightly.20260908.1400 against a real event store for a while, so orchestration_events holds a few million rows (mine: 4,019,125 events, 13.4 GB state.sqlite; others in Nightly 1400 backend exhausts heap during attachment cleanup on startup #10774 report 1.1M to 1.3M events at 7.9 GiB).
2. Update to 0.0.41-nightly.20260908.1400 or 1414.
3. Launch the desktop app.

Expected behavior

Migrations 48 and 49 run, the backend comes up, and the window loads existing threads.

Actual behavior

Migrations run, then the backend child replays the entire event log from sequence 0 for the new projection.attachment-cleanup projector. The V8 heap climbs to ~3.8 GB and the process aborts with FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory, about 20 to 80 s after spawn depending on store size. The desktop shell then logs Timed out after 60000ms waiting for desktop backend readiness and the app closes. It repeats on every launch. Three other users confirm the same in #10774, on 1400 and 1414.

Cause is already traced and a fix is open, but neither is in the triage queue:

• Diagnosis and repro data: Nightly 1400 backend exhausts heap during attachment cleanup on startup #10774 (no labels, filed outside the template)
• Fix: fix(server): release consumed event replay pages #10777 fix(server): release consumed event replay pages (CI green, no human review yet)

The projector was introduced by #9871. projection.attachment-cleanup is absent from the 0.0.36-nightly.20260828.1208 and 0.0.32 bundles and present in 1400/1414.

Impact

Blocks work completely

Version or commit

0.0.41-nightly.20260908.1414 (5d14c0e), also 1400

Environment

macOS 15.7.3 (24G419), Apple M1 Max, 64 GB RAM, arm64. Electron 44.1.0. Also reproduced by others on macOS 26.5.x with 24 GB.

Logs or stack traces

[11:02:00.573] INFO (#3): Migrations ran successfully
  migrations: [ '48_ProjectionThreadBranchPullRequest', '49_ProjectionThreadsActiveOrderKey' ]

<--- Last few GCs --->
[39184:0x120005b0000]    81863 ms: Mark-Compact (reduce) 3856.9 (3957.2) -> 3855.2 (3904.2) MB, pooled: 0.0 MB, 681.82 / 0.00 ms  (average mu = 0.198, current mu = 0.001) last resort; GC in old space requested
[39184:0x120005b0000]    82678 ms: Mark-Compact (reduce) 3855.2 (3904.2) -> 3855.1 (3902.2) MB, pooled: 0.0 MB, 814.35 / 0.00 ms  (average mu = 0.099, current mu = 0.000) last resort; GC in old space requested

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::OOMErrorHandler(char const*, v8::OOMDetails const&)
 6: v8::String::NewFromUtf8(...)
 7: node::sqlite::StatementExecutionHelper::ColumnToValue(...)
 8: node::sqlite::ExtractRowValues(...)
 9: node::sqlite::StatementExecutionHelper::All(...)
10: node::sqlite::StatementSync::All(v8::FunctionCallbackInfo<v8::Value> const&)

Timed out after 60000ms waiting for desktop backend readiness at http://127.0.0.1:3773/.well-known/t3/environment.

projection_state at time of crash: nine projectors at 4019125, projection.attachment-cleanup at 0 / 1970-01-01T00:00:00.000Z.

Workaround

Either works, neither is discoverable from the crash:

1. Roll back to the signed 0.0.41-nightly.20260908.1387 build. No DB edits needed; the 48/49 columns are harmless to it. Verified here: backend ready with 200, ~330 MB RSS, history intact.
2. Stay on 1414 and advance the cursor once, after backing up state.sqlite:

   UPDATE projection_state
   SET last_applied_sequence = (SELECT MIN(last_applied_sequence) FROM projection_state WHERE projector <> 'projection.attachment-cleanup')
   WHERE projector = 'projection.attachment-cleanup';

   Trade-off: attachments for already-deleted threads are never reclaimed.

NODE_OPTIONS=--max-old-space-size does not reach the backend child (Electron strips it, and the spawn uses execArgv: []).

Please consider pulling or flagging the 1400 and 1414 nightlies until #10777 lands, since the in-app updater still offers them.

[juliusmarminge]

Duplicate of #10774 — same Nightly 1400/1414 startup OOM while replaying the full event log for projection.attachment-cleanup.

Open fix: #10777 (fix(server): release consumed event replay pages).

[juliusmarminge]

Closing as duplicate of #10774.