Codex file-change diffs are duplicated in SQLite and exhaust backend heap

[alexfertel]

What happened

The T3 Code desktop backend still stops with a V8 out-of-memory error during Codex work. The UI then marks active work as failed and reports that the provider session did not survive a server restart.

This continued after the fix for #10924 shipped. That fix is present and limits provider event log records, but large Codex file-change diffs still enter durable SQLite activity data through a separate path.

Diagnosis

The fix from #12305 works for diagnostic provider logs. Post-fix provider log records on this install stay near the 64 KiB limit. The build that crashed also contains the #12305 merge commit.

The remaining unbounded path is durable activity persistence:

1. CodexAdapter.mapItemLifecycle attaches the full native lifecycle payload as payload.data.
2. ProviderRuntimeIngestion copies that data into both tool.started and tool.completed activities.
3. The resulting thread.activity-appended payload is stored in orchestration_events.
4. The projection pipeline stores the same activity payload again in projection_thread_activities.
5. Thread detail reads limit activity count to 500 and decode in batches of 25, but they do not limit bytes. projectActivityPayload drops unused diff data only after SQLite and V8 decode the full JSON.

The affected database is healthy but 15.9 GiB. It contains 12,551 file-change activity rows whose first changes[].diff field uses 5.2 GiB in each table, for 10.4 GiB across the event and projection copies. One stored diff has 49,359,322 characters. One activity row reaches 54.2 MiB. The newest 500 activities for the largest thread contain 571.7 MiB of raw JSON before decoding and projection.

The backend then aborts when V8 reaches its heap limit. Restarting clears live memory but leaves the large stored payloads, so the failure can return.

The current installed source still has this persistence path. A newer nightly exists, but no later commit has changed CodexAdapter, ProviderRuntimeIngestion, ProjectionSnapshotQuery, ActivityPayloadProjection, or the database migrations involved here.

Expected behavior: file-change lifecycle activities should persist bounded metadata such as paths, status, and omitted byte counts. They should not persist full diff text in both lifecycle rows and both SQLite stores. Existing oversized rows also need a supported repair or migration because the logger fix cannot clean stored data.

Steps to reproduce

Source-level reproduction:

1. Start the desktop app with a local backend and the Codex provider.
2. Send a valid Codex file-change item/started and item/completed lifecycle event whose item.changes[].diff contains a large string.
3. Let CodexAdapter map the event and let ProviderRuntimeIngestion create activities.
4. Inspect the stored thread.activity-appended event and its projected activity row.
5. Observe that the full diff exists in both orchestration_events.payload_json and projection_thread_activities.payload_json. Both lifecycle rows can retain the payload.
6. Repeat with tool-heavy work, or load a thread whose newest 500 activities contain several large file-change payloads.
7. Observe rapid database growth and eventual backend heap exhaustion while T3 processes the large activity window.

A focused regression test can use a generated diff string and assert that neither persisted copy contains the full string.

Version

Current install: 0.0.43-nightly.20260920.2005, commit 7445aa733ada.

Confirmed crashes: 0.0.43-nightly.20260918.1895, commit 994654198d58. This tag is 27 commits ahead of the #12305 merge commit, so the provider logger fix was present.

Environment

• T3 Code desktop app with local backend
• Darwin 25.6.0, arm64
• Node.js 26.8.2
• Codex provider

Evidence

SQLite quick_check: ok
Database size: 15,924.2 MiB

orchestration_events:              7,907.9 MiB
projection_thread_activities:      7,737.3 MiB

thread.activity-appended payloads: 7,705.3 MiB across 209,288 rows
tool.completed payloads:           4,372.8 MiB across 54,865 rows
tool.started payloads:             3,234.9 MiB across 54,914 rows

File-change diff fields in projection:
  rows:                             12,551
  total diff text:                  5,202.4 MiB
  largest diff:                     48.8 MiB

The event table contains the same row count and 5,202.4 MiB of diff text.

Largest complete activity row:     54.2 MiB
Largest newest-500 activity window: 571.7 MiB

Three backend crashes on the confirmed build:
  EXC_CRASH / SIGABRT
  FATAL ERROR: JavaScript heap out of memory
  Last-resort retained heaps ranged from about 1.6 to 1.95 GiB.

Post-fix provider log records remained bounded near 64 KiB.
No home paths, thread IDs, turn IDs, project names, commands, or payload text are included.

Related issues

• Oversized Codex diff event exhausts desktop backend heap #10924 and fix(server): bound provider event log records before serialization #12305 bound diagnostic provider log serialization. This report concerns the separate durable event and projection stores.
• [Bug]: Desktop backend still SIGABRTs with V8 OOM after 9–18h on 0.0.35 (hydration bounds already present) #8648 reports continued desktop backend OOM on large profiles. This report identifies a specific high-volume producer and the exact duplicate storage path.
• [Feature]: Add supported retention and compaction for unbounded local SQLite growth #5550 covers retention and compaction for unbounded SQLite growth. The affected install has no supported repair action for existing rows.
• [Bug]: Streaming tool.updated persists the full accumulated rawOutput on every delta — O(N²) bytes and rows per tool call #6673 and perf(server): persist the wire projection for streaming tool.updated data #6675 bound accumulated tool.updated persistence. tool.started and tool.completed still keep full file-change payloads.

Fix applied or workaround

No database write or local source patch was made. Restarting the desktop app lowers memory use for a time. Avoiding the affected heavy threads reduces the chance of another abort, but the oversized SQLite rows remain.

Filed by

Codex, GPT-5, via t3 triage

[juliusmarminge]

Triage

Confirmed against current main. This is a real backend crash, and it is not fixed by #12305.

#12305 only bounds diagnostic provider log records. The crashing builds already include that merge, and post-fix log rows staying near 64 KiB matches what we see in source. The remaining unbounded path is durable activity persistence.

What the code does

1. CodexAdapter.mapItemLifecycle attaches the full native lifecycle payload as payload.data, including Codex fileChange changes[].diff.
2. ProviderRuntimeIngestion copies that data into both tool.started and tool.completed. #6675 already persists the wire projection for streaming tool.updated and explicitly left terminal/start rows full.
3. thread.activity-appended is stored in orchestration_events, then written again to projection_thread_activities.
4. Thread-detail reads limit row count (500) and decode in batches of 25. They do not limit bytes.
5. projectActivityPayload drops unused diffs only after SQLite and V8 have decoded the full JSON. Clients only need paths/status from that projection.

That matches the measured install: a healthy 15.9 GiB DB, ~5.2 GiB of file-change diff text in each table, rows up to ~54 MiB, and a newest-500 window of ~572 MiB before decode. Restarting frees the heap and leaves the rows, so the abort can return.

#11598 storage cleanup does not rewrite these payloads.

Related, not duplicates:

• Oversized Codex diff event exhausts desktop backend heap #10924 / fix(server): bound provider event log records before serialization #12305 — logger bound only
• [Bug]: Streaming tool.updated persists the full accumulated rawOutput on every delta — O(N²) bytes and rows per tool call #6673 / perf(server): persist the wire projection for streaming tool.updated data #6675 — tool.updated persist slimming; this is the leftover started/completed hole
• perf(server): bound snapshot activity payload memory #9000 — snapshot batching; does not cap per-row bytes
• [Feature]: Add supported retention and compaction for unbounded local SQLite growth #5550 — retention/compaction; no supported payload repair
• [Bug]: Desktop backend still SIGABRTs with V8 OOM after 9–18h on 0.0.35 (hydration bounds already present) #8648 — broader desktop OOM on large profiles

Next step

Accept. Persist bounded file-change metadata (paths, status, omitted byte counts) at ingestion for tool.started and tool.completed, same idea as #6675. Add a regression that a generated oversized changes[].diff never appears in either SQLite copy.

Existing installs also need a supported rewrite + VACUUM. Restart cannot repair this. Rewriting orchestration_events is the part that needs a maintainer decision, because the event log is otherwise append-only.

Not closing. Not a logger-only or worktree-cleanup fix.