Skip to content

[Bug]: Codex goal turns run invisibly after a server restart (timeline stuck on "Thinking") #21865

New issue
New issue
Open
#25935
Open
[Bug]: Codex goal turns run invisibly after a server restart (timeline stuck on "Thinking")#21865
#25935
Labels
via-triageFiled through npx t3 triage

Description

@michal-jarczewski-deel
michal-jarczewski-deel
opened on Oct 19, 2026

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Start a Codex thread and set a goal with /goal …, so the goal is active and Codex continues turns on its own.
  2. Restart the T3 server while the goal is still active. In my case the desktop app's server restarted on its own about 45 minutes after the goal was set (server-runtime.json startedAt).
  3. Keep using the thread. I handed off to Claude for a few runs, then back to Codex with "continue".

Observed once, over ~20 hours, in a single thread. I haven't run a clean minimal repro.

Expected behavior

Goal continuation turns are attached to a run and render in the timeline. Failing that, the existing guard in the turn/started handler (orchestration-v2.codex-goal-turn-without-run: pause the goal and send turn/interrupt) stops them. Codex work shouldn't run for hours without any trace in the UI.

Actual behavior

After the restart, no Codex run in the thread gets any projected work items. The active run shows only "Working for 1h 23m" and "Thinking". Runs #16 and #17 show "Run interrupted before provider start". Run #19 stayed running for 2 hours with exactly one turn item (the user message).

Codex was working the whole time. In ~1.5 h it made 5 commits in the repo, and the provider log has ~1,200 events (commands, reasoning, agent messages, turn/started for new goal turns) on the main native thread. Subagent native threads were projected normally. Only the main thread's items were missing.

What the code and data suggest (server bundle, Codex adapter v2). This is inference; I didn't step through it in a debugger.

  • item/started / item/completed call awaitActiveTurn(payload.turnId) and return silently when no context exists, so every item of an unregistered turn is dropped.
  • The guard in turn/started requires rootProviderThreads.has(threadId). rootProviderThreads is in-memory only and is populated in runGoalCommand, i.e. only when /goal runs in the current server process. After a restart it is empty, so the guard never fires, the turn falls through to rememberSubagentTurnStarted, and its items are dropped. I found no codex-goal-turn-without-run warning in the server logs.
  • When T3 later sent turn/start for run Track per-thread terminal running state in UI and store #19, Codex returned an existing turn id (status: "inProgress", startedAt: null) that it had started on its own hours earlier. registerRootTurn was called with waitForNativeStart: true, but no new turn/started for that id arrived. orchestration_v2_projection_provider_turns has no rows for this provider thread after run oxc stack #1.

Workaround that worked once: quit and relaunch the app. This also stops the old codex app-server. The next Codex run rendered normally (messages, commands, browser use, tasks), and /goal wasn't needed. I haven't confirmed whether later goal continuation turns in the new server process stay visible.

Possible fixes:

  • Rehydrate goal state on server start or provider-thread resume (e.g. thread/goal/get for root provider threads), or persist rootProviderThreads.
  • When turn/start returns an already in-progress turn, register it right away instead of waiting for a turn/started that never comes.
  • At minimum, log a warning instead of silently returning when item events arrive for an unknown turn on a root provider thread.

Related, but not the same trigger: #20232 (native goal work continues behind a failed run after /goal resume), #18833, #16942.

Impact

Major degradation or frequent failure

Version or commit

v0.1.0 (stable desktop release)

Environment

macOS (Darwin 25.2.0, arm64), T3 Code desktop app with local server; Codex provider gpt-6.1-sol, Claude Agent provider claude-opus-5-5 in the same thread via context handoff

Logs or stack traces

# provider log, main native thread, run #19 start (UTC)
16:25:41.094 outgoing thread/inject_items   (context handoff)
16:25:41.112 outgoing turn/start            input=[{text:"continue"}]
16:25:41.115 incoming result turn.id=<T>    status=inProgress startedAt=null   # <T> was first started 18 h earlier
16:25:59.567 incoming item/started userMessage "continue" turnId=<T>
...          ~1,200 item/* and turn/started events on the main thread over the next 1.5 h

# state DB (copy), run #19
orchestration_v2_projection_turn_items  where run_id = run #19  -> 1 row (user_message)
orchestration_v2_projection_provider_turns for this provider thread -> last row from run #1

Activity

  1. juliusmarminge commented on Oct 19, 2026

    @juliusmarminge
    juliusmarminge
    on Oct 19, 2026
    Member

    Note

    Grok responding on behalf of Julius.

    Thanks for the detailed log excerpts. I checked the diagnosis against current main, and two parts don't match it.

    • The list of root provider threads is in memory only (apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:1904). It is filled every time T3 registers a root turn (CodexAdapterV2.ts:1931-1938, :2046), not only when /goal runs, so it doesn't stay empty after a restart.
    • When turn/start returns a turn that hasn't started yet, the turn is registered right away and its items can attach to it (CodexAdapterV2.ts:1984-2024, :6225-6230). Only a Stop waits for the native start.

    Items for a turn T3 never registered are dropped without a log line (CodexAdapterV2.ts:2058-2080, :4766-4769). That part matches, but it doesn't explain why your registered turn showed nothing.

    Since this happened once on 0.1.0, could you share whether it happens again on the latest nightly? If it does, the server log from around the restart and the first run after it would help most.

  2. juliusmarminge
    added
    needs more infoInitial triage showed no bug. Awaiting more info
    via-triageFiled through npx t3 triage
    and removed
    needs more infoInitial triage showed no bug. Awaiting more info
    on Oct 19, 2026

  3. michal-jarczewski-deel commented on Oct 29, 2026

    @michal-jarczewski-deel
    michal-jarczewski-deel
    on Oct 29, 2026 – with ChatGPT Codex Connector
    Author

    Thanks Julius. I haven't confirmed another occurrence on my current desktop install (0.1.5), and I haven't run a controlled repro on the latest nightly.

    Verified from the retained provider logs for the original incident (all times UTC):

    2026-10-18
19:52:59.124  thread/resume sent
19:53:00.036  resume response received
19:53:00.074  turn/start sent
19:53:00.086  thread/goal/updated: active
19:53:00.087  turn/started received
19:53:00.087  turn/start failed: "failed to submit turn input: EmptyInput"

    Later, for run #19 on October 19, the logs show thread/inject_items at 16:25:41.094, turn/start at 16:25:41.112, and its response at 16:25:41.115 returning an existing turn with status: "inProgress" and startedAt: null. That turn's original turn/started was at 22:04:19.660 the previous day.

    The earlier resume/goal/EmptyInput sequence looks consistent with the failure described in your PR #25935. Perhaps registering the root before resume will prevent the chain that led to the later invisible run, but that is still an inference.

    We need to wait for #25935 to be merged and available in a build, then retest the original macOS restart → Claude → Codex “continue” sequence before confirming this issue is fixed. The existing-in-progress-turn response above is the additional case to check.

  4. costaspanay commented on Nov 11, 2026

    @costaspanay
    costaspanay
    on Nov 11, 2026 – with ChatGPT Codex Connector

    Confirmed another occurrence on T3 Code 0.1.9-nightly.20261111.4501, macOS 27.0 (26A428), Codex app-server 0.167.1. This follows an automatic server-restart continuation, rather than an explicit /goal resume. It combines the invisible execution described here with the EmptyInput failure in #20232.

    Observed sequence

    An existing long-running Codex thread had an active goal. The local server restarted at 2026-11-11 05:15:44.547 UTC (server-runtime.json.startedAt). T3 projected a “T3 Code restarted and resumed this turn” notification, followed by the generic “The provider could not start this turn” error.

    Two automatic continuation attempts failed, followed by three manual continue attempts. Changing the selected model from gpt-6-astra to gpt-6.1-sol and back did not recover visible execution. All five T3 runs (ordinals 75–79) were persisted as failed.

    Despite that failed UI state, the same native Codex turn continued receiving the manual messages and executing commands and file changes. The provider event log records work after the latest displayed error, while T3's thread activity contains only the failed runs and no corresponding new work items.

    Sanitized protocol evidence

    Selected fields from the affected thread's provider event log; all timestamps UTC on 2026-11-11. <N> is the same native thread throughout and <T> is the same native turn throughout. Identifiers, workspace paths, credentials, task content and unrelated payloads are omitted.

    05:15:46.508 outgoing thread/resume threadId=<N> model=gpt-6-astra
05:15:47.389 incoming thread/status/changed status=idle
05:15:47.457 outgoing turn/start threadId=<N> input=[]                 rpc=3
05:15:47.492 incoming thread/status/changed status=active
05:15:47.492 incoming turn/started turn.id=<T> status=inProgress
05:15:47.493 incoming error rpc=3 code=-32603
                      message="failed to submit turn input: EmptyInput"

05:16:48.261 outgoing turn/start threadId=<N> input=[]                 rpc=5
05:16:48.263 incoming error rpc=5 code=-32603
                      message="failed to submit turn input: EmptyInput"

05:19:48.365 outgoing turn/start input=[{text:"continue",type:"text"}]  rpc=7
05:19:48.369 incoming result rpc=7 turn.id=<T>
                      status=inProgress startedAt=null
                      # T3 run 77 is marked failed

05:20:23.027 outgoing thread/resume threadId=<N> model=gpt-6.1-sol
05:20:23.077 outgoing turn/start input=[{text:"continue",type:"text"}]  rpc=10
05:20:23.081 incoming result rpc=10 turn.id=<T>
                      status=inProgress startedAt=null
                      # T3 run 78 is marked failed
05:20:59.203 incoming item/started turnId=<T> type=commandExecution

05:21:05.887 outgoing thread/resume threadId=<N> model=gpt-6-astra
05:21:05.926 outgoing turn/start input=[{text:"continue",type:"text"}]  rpc=13
05:21:05.931 incoming result rpc=13 turn.id=<T>
                      status=inProgress startedAt=null
05:21:05.933 T3 run 79 completed with status=failed

05:21:06.039 incoming item/started turnId=<T> type=userMessage
05:21:34.172 incoming item/started turnId=<T> type=commandExecution
05:22:28.988 incoming item/started turnId=<T> type=commandExecution
05:22:28.994 incoming item/started turnId=<T> type=fileChange
05:22:29.065 incoming item/started turnId=<T> type=commandExecution

    A read-only inspection of orchestration_v2_projection_run_attempts confirmed that the latest attempt had status:"failed", the correct nativeThreadId, and providerTurnId:null. T3's thread read returned status:"failed" and activeRunId:null even though the provider log showed continued native execution.

    Reproduction and verification limits

    This is evidence from the affected live session, not a clean deterministic reproduction. The observed path was:

    1. Run a Codex thread with an active goal.
    2. Restart the T3 server and allow its automatic continuation.
    3. Observe turn/start input=[], EmptyInput, and the generic failed-run banner.
    4. Send continue; repeat with a different Codex model selected.
    5. Compare T3's failed run/timeline state with continuing native item/* events for the original turn.

    No provider restart, goal clear, database edits or recovery experiment was performed during diagnosis. The cause of the server restart was not established. The adapter's exact internal reason for marking the nonempty retries failed has not been established.

    Expected recovery / restoration check

    The automatic continuation should not leave an untracked active native turn after EmptyInput. When a nonempty retry returns an already in-progress native turn, T3 should either attach and render it correctly or stop/reconcile it explicitly. It must not report an idle failed thread while that thread's native agent continues editing files.

    A useful E2E check would cover restart during an active goal, the empty-input failure, a subsequent nonempty retry, and visible command/file-change events matching the native turn. Switching models should not be required for recovery.

    Related: #20232 and the still-open proposed fix #25935. #17589 is closed without being merged; it should not be treated as a shipped recovery fix.

    Prepared with GPT-6.1-Sol through Codex in T3 Code, at the affected user's request. No full transcript or raw log upload is included.

  5. juliusmarminge
    added a commit that references this issue on Nov 11, 2026

    test(server): wait on the interrupt request instead of wrapping notif…

    d1401a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions