[Bug]: In-memory ingestion backlog is lost on restart, bricking threads in "starting"

[kelchm]

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

Observed in production use on 2026-08-13 (timeline below). A deterministic reproduction should be possible with these steps:

1. Flood provider runtime ingestion until its worker falls far behind—for example, with the Grok tool_call_update storm in [Bug]: Grok ACP resends full cumulative terminal output in every tool_call_update, flooding ingestion for all threads #6556 or the Codex subagent progress flood in [Bug]: Codex subagent progress floods orchestration ingestion and delays unrelated threads #5681.
2. While ingestion is backlogged, send a message to a healthy thread on another provider (Codex in the incident) and let the provider complete the whole turn. The canonical events are published, and CheckpointReactor even captures a checkpoint, but ingestion has not processed them yet.
3. Quit or restart the app before ingestion catches up.
4. After restart, open the thread and try the stop/interrupt button.

Expected behavior

Either the canonical runtime event stream should be durable enough to replay after a restart, or startup should reconcile threads whose projections show a pending or running turn with provider reality. The session directory still contains the resume cursor, and the provider-side thread shows that the turn completed.

At minimum, interrupting a thread with no active turn should clear the pending turn-start and return the session to ready, allowing the user to escape the stuck state.

Actual behavior

ProviderRuntimeIngestion.ts consumes providerService.streamEvents through a single serial DrainableWorker backed by an unbounded, in-memory TxQueue. Ingestion stopped dispatching at 03:26:17 — whether the worker was wedged behind the backlog or its subscription had already died, the unprocessed events existed only in this in-memory pipeline, so the server shutdown (provider.stopAll at 03:39:08) discarded them.

The canonical events exist nowhere else in orchestration state; they remain only in the debug NDJSON provider log. The loss is therefore permanent:

• The thread's two completed Codex turns never produced assistant thread.message-sent events or the running/ready thread.session-set transitions — only the client message-sent and the initial "starting" session-set from each turn request exist. Both assistant replies are still missing from the thread.
• The session projection remains stuck at status = "starting", with a leaked pending turn-start row: a projection_turns row with a NULL turn_id.

There is no interrupt- or startup-driven escape from this state:

1. No startup reconciliation — The restarted server never re-examines the pending turn-start. This is the same gap described in [Bug]: App restart during active turns leaves threads stuck on Working forever, with no startup reconciliation of dead provider sessions #4584 / proposal Proposal: recover stuck running turns when turn/completed is lost after long command execution #917. The thread only "recovers" if the user happens to send a new message.
2. Interrupt is a no-op — processTurnInterruptRequested (ProviderCommandReactor.ts:1187) only calls providerService.interruptTurn. During the incident, this actually recovered the Codex session (session/connecting → session/ready, MCP servers ready), but there was no active turn to interrupt and nothing cleared the pending state.
3. The session cannot leave "starting" — Ingestion deliberately maps session/ready back to "starting" while a pending turn-start exists (ProviderRuntimeIngestion.ts:1564). As a result, even the successful session recovery in step 2 left the UI spinner running forever.

Incident timeline (UTC, 2026-08-13), reconstructed from orchestration_command_receipts, orchestration_events, provider_session_runtime, and the provider NDJSON logs:

03:22–03:26  Grok tool_call_update storm floods ingestion (#6556)
03:26:17     last ingestion dispatch (receipts with `provider:` command-id prefix stop)
03:26:48     user message → Codex thread; turn completes 03:29:12 (canonical events
             published; checkpoint captured by CheckpointReactor) — never ingested
03:36:50     second user message; turn completes 03:37:49 — same fate
03:39:08     server shutdown (provider.stopAll) — in-memory ingestion backlog discarded
03:41:09     server back; ingestion dispatches resume for OTHER threads' fresh events only
04:16:37     user hits interrupt: Codex session reconnects and reports ready, but the
             projection stays "starting"; second interrupt is a silent no-op. Frozen.

This is distinct from #5781, where the adapter consumer fiber dies and events never reach the bus. Here, the events demonstrably reached the bus: CheckpointReactor consumed them and captured checkpoints for turns whose messages were never persisted. That inconsistency also provides a useful signal for a reconciler.

Impact

Major degradation or frequent failure

Version or commit

Desktop 0.0.33 / main @ e5c82d7

Environment

macOS 26.5.2 (Darwin 25.5.0), Codex app-server provider (gpt-5.6-sol) on the affected thread, Grok CLI 1.0.3 as the flood source

Logs or stack traces

All excerpts below are from `~/.t3/userdata/state.sqlite` and `~/.t3/userdata/logs/provider/events.3540be8c-….log`, lightly annotated, with long lines truncated at `…`. State rows were captured ~2026-08-13T04:30Z, while the thread was still frozen.

Command receipts for the affected thread, 03:26–04:17. Note the total absence of `provider:`-prefixed receipts (runtime ingestion dispatches) between 03:26:48 and 04:16:37, while `server:checkpoint-*` receipts land at 03:29:12 and 03:37:49 for the very turns whose messages were never persisted:


sqlite> SELECT command_id, accepted_at FROM orchestration_command_receipts
   ...> WHERE aggregate_id='3540be8c-edc6-4d96-9460-decfbf8001ea'
   ...>   AND accepted_at BETWEEN '2026-08-13T03:26' AND '2026-08-13T04:17' ORDER BY accepted_at;
e89cdc52-2416-44b9-a089-8a4d54c1ec02|2026-08-13T03:26:48.012Z                              -- user turn-start
server:provider-session-set:a9f5d624-1045-4a7a-803d-e15c48ff75c3|2026-08-13T03:26:48.012Z
server:checkpoint-turn-diff-complete:b3fb918e-7144-4147-b28b-23d6c7225049|2026-08-13T03:29:12.794Z
server:checkpoint-captured-activity:cd9dd77b-3125-4054-9ebf-6e43976608f2|2026-08-13T03:29:12.794Z
7118e14f-ef65-48f9-a5b8-550104e312ac|2026-08-13T03:36:50.279Z                              -- user turn-start
server:provider-session-set:606ead09-d1d4-488a-889a-e3e17f709e7c|2026-08-13T03:36:50.279Z
server:checkpoint-turn-diff-complete:c160cba2-c6c0-4e6f-a463-34b1127f6cd6|2026-08-13T03:37:49.707Z
server:checkpoint-captured-activity:77640006-c6b5-401e-b93f-20121c692e8e|2026-08-13T03:37:49.707Z
741930ea-92a2-4eea-bd1b-8837c1427229|2026-08-13T04:16:37.821Z                              -- user interrupt
provider:df1e199f-2425-44dd-9701-6c1283f5aaed:thread-session-set:1412d1de-…|2026-08-13T04:16:37.847Z
provider:9b3f44ce-0f90-4221-b683-ca85baf2714f:thread-session-set:98f272c0-…|2026-08-13T04:16:37.969Z
provider:03c8a382-0c91-409c-8e48-e13ddd66a435:thread-activity-append:4982f2a4-…|2026-08-13T04:16:37.970Z
501b5b4a-5a05-42b5-8e33-2f679e27ad30|2026-08-13T04:16:38.993Z                              -- second interrupt: no receipts follow


State rows while frozen — session directory says the session is fine (recovered, resume cursor intact, last event `provider.stopAll` at shutdown), but the projection is stuck `starting` with a leaked pending turn row:


sqlite> SELECT * FROM provider_session_runtime WHERE thread_id='3540be8c-…';
3540be8c-edc6-4d96-9460-decfbf8001ea|codex|codex|auto|running|2026-08-13T04:16:37.969Z|{"threadId":"019fbbac-1f21-7953-96c9-6e67bd93a5cf"}|{"cwd":"/Users/kelchm/.t3/worktrees/home-lab/t3code-6046b50c","model":"gpt-5.6-sol","activeTurnId":null,"lastError":null,…,"lastRuntimeEvent":"provider.stopAll","lastRuntimeEventAt":"2026-08-13T03:39:08.912Z"}|codex

sqlite> SELECT * FROM projection_thread_sessions WHERE thread_id='3540be8c-…';
3540be8c-edc6-4d96-9460-decfbf8001ea|starting|codex|||||2026-08-13T04:16:37.969Z|auto|codex

sqlite> SELECT turn_id, state, requested_at, started_at, completed_at FROM projection_turns
   ...> WHERE thread_id='3540be8c-…' ORDER BY requested_at DESC LIMIT 2;
019ff931-c406-7f42-af01-ae5b90c6543a|completed|2026-08-13T03:37:49.707Z|2026-08-13T03:37:49.707Z|2026-08-13T03:37:49.707Z
|pending|2026-08-13T03:36:50.279Z||


Provider NDJSON log proves the turn completed and the reply text exists — it just never reached orchestration:


[2026-08-13T03:37:49.712Z] NTIVE: {"id":"b04f5a5f-72c5-4dcc-85f9-5bd859e8483f","provider":"codex","providerInstanceId":"codex","createdAt":"2026-08-13T03:37:49.707Z","kind":"notification","threadId":"3540be8c-edc6-4d96-9460-decfbf8001ea","method":"turn/completed","turnId":"019ff931-c406-7f42-af01-ae5b90c6543a","payload":{"threadId":"019fbbac-1f21-7953-96c9-6e67bd93a5cf","turn":{"completedAt":1786592269,"durationMs":53518,"error":null,…

[2026-08-13T03:37:49.712Z] NTIVE: {"id":"efcc8fa0-7b91-49cf-a3ae-62bb0bb99251",…,"method":"item/completed","turnId":"019ff931-c406-7f42-af01-ae5b90c6543a","itemId":"msg_039b496747c19434016a7d3c079e888195bb20b71e97ad3fda","payload":{"completedAtMs":1786592269433,"item":{…,"phase":"final_answer","text":"Going well. Current state:\n\n- [Draft PR #306](https://gi…

[2026-08-13T03:37:49.719Z] CANON: {"eventId":"b04f5a5f-72c5-4dcc-85f9-5bd859e8483f","provider":"codex","threadId":"3540be8c-edc6-4d96-9460-decfbf8001ea","createdAt":"2026-08-13T03:37:49.707Z","turnId":"019ff931-c406-7f42-af01-ae5b90c6543a","providerRefs":{"providerTurnId":"019ff931-c406-7f42-af01-ae5b90c6543a"},"raw":{"source":"codex.app-server.notification","method":"turn/completed",…

Screenshots, recordings, or supporting files

No response

Workaround

Send a new message to the frozen thread. The new turn-start supersedes the stale pending one, and the recovered session handles it — confirmed on the affected thread roughly 21 hours later: the first new message immediately started a turn that ran and completed normally. The lost assistant replies can be recovered manually from ~/.t3/userdata/logs/provider/events.<threadId>.log; the item.completed agentMessage entries contain the full text.

[joshfcc]

Independent repro on macOS with T3 Code Nightly 0.0.34-nightly.20260817.1113.

Affected T3 thread: 3bbf530c-b3a3-4756-913f-fe9064c2d1f7

The provider turn completed successfully at 2026-08-17T05:11:38.759Z and the checkpoint was persisted. The final assistant response is still recoverable in the provider NDJSON log. However, no assistant thread.message-sent event or completed session lifecycle transition was projected.

State after restart:

• projection_turns: completed
• provider_session_runtime: stopped
• projection_thread_sessions: starting
• pending_messages: orphan row with a null turn ID
• projection_thread_messages: user prompt only, no assistant message
• UI: still shows Working with only the prompt visible

The restart dropped the in-memory ingestion backlog and left the thread internally contradictory. The source flood in this run is documented in #5681. Runtime events adjacent to this thread checkpoint were about 31 minutes old when the checkpoint was persisted.

This looks like the same durability boundary described here, with a concrete case where the provider result survived on disk but the canonical projection did not.