Skip to content

feat(orchestrator): Surface waiting background work - #4378

Merged
juliusmarminge merged 1 commit into
pingdotgg:t3code/codex-turn-mappingfrom
mwolson:feat/orchestrator-v2-background-waiting
Aug 5, 2026
Merged

feat(orchestrator): Surface waiting background work#4378
juliusmarminge merged 1 commit into
pingdotgg:t3code/codex-turn-mappingfrom
mwolson:feat/orchestrator-v2-background-waiting

Conversation

@mwolson

@mwolson mwolson commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Show Waiting after a root turn settles while finite provider background work
    remains.
  • Normalize command, dynamic-tool, subagent, and Claude SDK background-task
    lifecycle into one per-thread roster.
  • Present Waiting consistently in the web timeline, classic sidebar, SidebarV2,
    mobile thread feed, and classic and native mobile lists while preserving
    Working for an active turn.
  • Keep provider ingestion and idle release alive until finite background work
    clears, then persist the empty roster that returns the thread to Ready.
  • Replay a recorded Claude background-Bash transcript through the full
    OrchestratorV2 pipeline with a deterministic frame gate to pin Waiting after
    root settlement, late completion, roster clearing, and return to Ready.
  • Hide mobile Stop once the runtime is parked at idle, preserving Send while
    background work finishes.

Presentation Scope

#4415 waiting-presentation
closed unmerged, so it is no longer a dependency of this PR. This branch now
contains the orchestrator-v2 roster contract plus the web and mobile
presentation listed above.

The native Thread List v2 is now the default mobile list. It consumes the same
parked-idle Waiting mapping, so its rows stay consistent with Sidebar V2 and
the classic mobile list.

Problem and Fix

Problem and Why it Happened Fix
A completed root turn can still own finite background work, but the v2 UI had no normalized post-settlement signal and could look idle. Derive normalized pending background work from existing projected turn items and Claude's authoritative SDK task roster.
The visible classic web sidebar ignored that normalized roster and fell back to a timestamp. Render a static, muted Waiting pill from pendingBackgroundTasks, after active Working and Connecting precedence.
Closing the provider event subscription as soon as a root run settled could miss the final task-roster update. Keep the subscription open while that provider thread still has finite background work, then persist the empty roster that clears Waiting.
A session-wide pending-work probe could couple unrelated provider threads on a shared runtime. Use a thread-scoped probe for run ingestion while retaining the session-wide probe for idle release.
Existing stored thread shells do not contain the new field. Decode pendingBackgroundTasks with an empty-array default for backward compatibility.
The mobile conversation view had no Waiting row at all. Nothing under apps/mobile called derivePendingBackgroundWork, and the view prefers deriveThreadRuntime(projection), which has no roster parking, so activeWorkStartedAt went null at settlement and the working row simply vanished. deriveThreadPendingBackgroundWork in apps/mobile/src/lib/threadActivity.ts derives the roster from the projection and deriveThreadFeedPresentation appends a waiting-background row. It is a pure helper rather than inline hook code so the policy is unit tested; apps/mobile has no hook-rendering test infrastructure.
The active classic mobile thread list showed no status because resolveThreadStatus had no Waiting branch. resolveThreadStatus gained a static muted Waiting pill, ranked below approvals, input, active work and failures, and above Plan Ready.
The default native Thread List v2 mapped a parked runtime.idle shell to Ready and had no Waiting label. Map parked idle to a muted Waiting row while preserving approval, input, active work, failure, and quiescent Ready precedence.

Defensive Fixes

Problem and Why it Happened Fix
Claude SDK roster levels and completion edges can arrive in either order. Track authoritative roster state separately from wake eligibility and replay classification so Waiting clears without losing exactly-once continuation.
A finite Monitor can emit a terminal notification without Claude starting a native wake turn. Buffer opaque notifications for replay, but request their continuation only after native user, assistant, or result output proves Claude actually began the wake.
Replacing a Claude query kills the old CLI process before it can clear its roster. Clear the dead thread's process-scoped roster and wake state, preserve only explicitly classified buffered Bash notifications for same-thread replacement, and keep buffered subagent notifications on the subagent lifecycle.
A Grok subagent can outlive the root turn through adapter carryover. Include live carryover subagents in the ACP pending-work probe so idle release cannot kill work that is still projected as Waiting.
Persisted background state can survive a process restart without a provider capable of completing it. Clear persisted rosters, cancel stale background-capable items, and attribute recovery events to the run, subagent, or matching provider thread.
Mobile restores projections from disk, and shouldPersistThread persists only settled ones, so a cached projection is exactly the shape that reproduces a Waiting row for work that already finished. Web cannot hit this. The mobile roster is gated on a live thread detail. synchronizing is refused alongside cached, because a subscription passes through it before any fresh projection arrives, so its data can still be the cached one.
The shared helper's runs argument is optional, and omitting it leaves the rolled-back-run exclusion set empty, so mobile would have shown Waiting for turn items owned by a rolled_back run where desktop shows nothing. Mobile passes runs: projection.runs, matching the web call site argument for argument. A focused test pins it: deleting that one argument fails exactly that test and nothing else.
formatPendingBackgroundWorkLabel interpolates the raw description, which for a subagent item is the entire child prompt, wrapping into a wall of text on a phone. The mobile label is clamped to two lines.
A successful run persists as waiting with a null completedAt until checkpoint capture lands, so deriveActiveWorkStartedAt kept reporting a start time and the feed's Working branch hid Waiting for that whole window, which is exactly when Waiting should first appear. Found by Bugbot on the mobile commit. deriveThreadWorkingStartedAt suppresses the start time once the roster has parked the runtime at idle, which is how web sequences Working against Waiting. A waiting run with an empty roster still reads Working, since that is checkpoint capture with no background work behind it. A regression test pins the parked case.
A stale activeRunId on a runtime already parked at idle could leave mobile Stop armed and replace Send. Require both an active runtime status and a non-null interruptible run id. preparing and starting remain interruptible, while parked Waiting exposes Send.

Manual test guide

Browsable scenario guide: https://nam7nt0rbtm6.postplan.dev/

This PR owns the Waiting state, so most background scenarios in the guide assert
it. The ones that matter, across all three providers:

Waiting must appear

Waiting must NOT appear. These are the false-positive guards, and they are
the easier half to regress:

Related follow-up

Validation

  • Focused server, contract, shared, client-runtime, web, mobile, and Stop-guard
    suites pass.
  • The focused native Thread List v2 suite passes all 37 tests, including
    parked-idle Waiting, quiescent Ready, and approval precedence.
  • The recorded claude_background_task_after_root replay fixture drives the
    real adapter and OrchestratorV2 pipeline. It pins the nonempty roster, root
    Waiting state, late empty roster, provider return to idle, and final run
    completion without a wall-clock delay. Four replay and contract files pass 88
    tests, and the focused fixture passed 10 consecutive stress runs after
    cancellation and mismatch cleanup review.
  • vp check and git diff --check pass. Full typecheck reaches only the known
    untouched CTM baseline error at
    apps/server/src/httpCompression/HttpResponseCompression.ts:33.
  • Real-provider experiment packs passed for Claude background Bash and
    subagents, Codex background exec and subagents, and Grok continuations and
    subagents.
  • Isolated integrated web verification with a real Claude background Bash task:
    the classic sidebar row displayed static, muted Waiting after root settlement,
    then returned to its timestamp after the roster cleared.
  • A deterministic private-server Opus replay reproduced a nested finite Monitor
    that ended with notification-only traffic. Waiting cleared after run 2, no
    empty run 3 appeared, and the provider stayed idle through a 35-second
    residual window.
  • Packaged in-vivo round against the built AppImage covering the
    notification-only path and its regression guards. Two independent scenarios
    drove a real nonpersistent Monitor whose terminal notification arrived with no
    native wake frames: both cleared Waiting without opening a run, and each held
    at two completed runs, idle with an empty roster, through a residual window of
    about 176 seconds. The prior defect left an empty run active for 225 seconds.
  • Regression guards passed in the same round: background Bash still wakes exactly
    once across a separate follow-up turn, a resumed subagent still wakes, and Stop
    during a blocking TaskOutput still leaves two distinct clean recovery turns.
    Every wake log marker in the round was attributable to an expected wake, with
    none unattributed.
  • Packaged classic-sidebar verification against the running desktop app's own web
    server: the row rendered a static muted Waiting pill with dot class
    bg-sidebar-muted-foreground and no pulse while the roster was nonempty, and
    title plus timestamp only after it cleared. An adjacent thread rendered Working
    in the same view, confirming the states are visually distinct.
  • On the current published head, Release Smoke, Mobile Native Static Analysis,
    Cursor Bugbot, and both Macroscope code checks pass. Check is red only because
    Vite+ panicked while printing lint warnings with Resource temporarily unavailable; local vp check passes. Test is red only for the two known CTM
    ProjectScriptsControl w-7 assertions. Vercel remains red for fork
    authorization.

This branch is based on its approved t3code/codex-turn-mapping snapshot.


Note

High Risk
Changes orchestration runtime idle/release, ACP post-settle carryover, and MCP status semantics—areas where subtle race or double-projection bugs affect live agent threads and interrupt UX.

Overview
Surfaces Waiting when a thread is parked at runtime.idle but still has post-settlement background work, aligned with the web sidebar: mobile Thread List v2 adds a muted Waiting status (mapped from idle + pending roster) and the composer only treats runs as interruptible via threadRuntimeHasInterruptibleRun, so Stop stays hidden while waiting and Send remains available.

Orchestrator MCP thread list/read/filtering now prefer activityRunStatus over shell status so a still-active run is not masked by a newer cancelled queued run; taskStatus returns the task’s real providerInstanceId instead of deriving it from the driver kind.

ACP adapter gains tighter post-settle continuation and carryover behavior (hasPendingBackgroundWork pinning, deferred terminal projection on attach, wake-buffer ownership when user turns race continuations), with a large new effect/live test suite covering those paths plus continuation helper predicates.

Reviewed by Cursor Bugbot for commit b2c68ba. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Surface post-settlement background work as a 'Waiting' state in the sidebar and chat timeline

  • Adds a pendingBackgroundTasks roster to OrchestrationV2ProviderThread and OrchestrationV2ThreadShell contracts, populated by Claude adapter per-native-thread background task tracking and derived client-side via a new derivePendingBackgroundWork shared utility.
  • When a settled run has pending background tasks, shellRuntime and deriveThreadRuntime park the runtime status at 'idle'; the sidebar resolves this as 'waiting' and the chat timeline emits a new waiting-background row with a pulsing indicator.
  • Adds writeIfProviderThreadOwner to EventSinkV2 for ownership-gated provider thread writes, and extends RunExecutionService to route inherited background turn_item.updated events from eligible prior interrupted/failed/cancelled runs.
  • ProviderRuntimeRecoveryService now cancels stale nonterminal background-capable turn items on restart and clears persisted pendingBackgroundTasks rosters on all provider threads.
  • ACP adapter gains pending subagent status as a settlement blocker, terminalStatusProjected tracking to prevent double-emission of carryover terminals, and hasPendingBackgroundWork pinning until all active or carryover subagents with unprojected terminals are resolved.
  • Adds a ProviderReplayGate testkit primitive and release_replay_gate / capture_shell_snapshot scenario steps to enable fixture-driven testing of background task lifecycle.
  • Risk: threadRuntimeHasInterruptibleRun now returns false when runtime status is idle even if activeRunId is set, which suppresses the interrupt button for parked-idle threads.

Macroscope summarized b2c68ba.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6617bb9e-32c5-4890-90c6-7ea892edf628

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Jul 23, 2026
Comment thread apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from e46af0a to f318877 Compare July 24, 2026 04:04
Comment thread apps/server/src/orchestration-v2/RunExecutionService.ts
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 2 times, most recently from 1e58e65 to a286c60 Compare July 24, 2026 13:37
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from f318877 to 2bf4cdf Compare July 24, 2026 15:32
Comment thread apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch 7 times, most recently from 4f72b01 to dcb197d Compare July 25, 2026 04:10
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 25, 2026
Replays the amended PR pingdotgg#4378 wake-settlement fix onto the v2.1 trial. A
terminal task notification no longer opens an opaque-task continuation
unless native user, assistant, or result output proves Claude began the
wake turn.
@mwolson
mwolson marked this pull request as ready for review July 25, 2026 13:58
Comment thread apps/server/src/orchestration-v2/ProjectionStore.ts Outdated
Comment thread apps/server/src/orchestration-v2/RunExecutionService.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

You can customize Macroscope's approvability policy. Learn more.

@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from bd45aae to 29cfba8 Compare July 25, 2026 15:08
Comment thread apps/web/src/components/ChatView.tsx
Comment thread packages/shared/src/orchestrationV2PendingBackgroundWork.ts
Comment thread packages/shared/src/orchestrationV2PendingBackgroundWork.ts
Comment thread apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from 123245f to fc4cc2c Compare July 26, 2026 00:26
Comment thread apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from fc4cc2c to 2f07ff6 Compare July 26, 2026 00:57
Comment thread apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts
Comment thread apps/web/src/components/ChatView.tsx
Comment thread apps/mobile/src/lib/threadActivity.ts Outdated
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from e153ab5 to 5572efb Compare July 30, 2026 01:45
Comment thread apps/web/src/components/Sidebar.logic.ts
Comment thread packages/shared/src/orchestrationV2PendingBackgroundWork.ts
Comment thread apps/mobile/src/state/use-thread-composer-state.ts Outdated
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from 5572efb to 8024ba3 Compare July 30, 2026 02:03
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 30, 2026
…ntime status

The CTM merge left two pingdotgg#4415 sidebar-waiting-state assertions that cannot hold
on this line, because CTM keys thread status off runtime where main uses
session:

- apps/mobile threadListV2.test.ts asserted Waiting through session.idle.
  EnvironmentThreadShell has no session field here, so it did not typecheck.
  pingdotgg#4378 orchestrator-v2-background-waiting already covers the same behaviour
  through the presentation bridge, so the duplicate goes.
- apps/web Sidebar.logic.test.ts asserted that an idle runtime carrying a
  stale lastError resolves to ready. On this line runtime.idle is Waiting: the
  pingdotgg#4378 bridge parks a completed shell at idle only when the post-settlement
  background roster is nonempty. The sibling completed-status assertion still
  covers the stale-lastError case.
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 30, 2026
Carry the append-only review fixes from pingdotgg#4378 orchestrator-v2-background-waiting into pingdotgg#4547 orphaned-output-after-early-settle without rewriting reviewed history.
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 30, 2026
…ntime status

The CTM merge left two pingdotgg#4415 sidebar-waiting-state assertions that cannot hold
on this line, because CTM keys thread status off runtime where main uses
session:

- apps/mobile threadListV2.test.ts asserted Waiting through session.idle.
  EnvironmentThreadShell has no session field here, so it did not typecheck.
  pingdotgg#4378 orchestrator-v2-background-waiting already covers the same behaviour
  through the presentation bridge, so the duplicate goes.
- apps/web Sidebar.logic.test.ts asserted that an idle runtime carrying a
  stale lastError resolves to ready. On this line runtime.idle is Waiting: the
  pingdotgg#4378 bridge parks a completed shell at idle only when the post-settlement
  background roster is nonempty. The sibling completed-status assertion still
  covers the stale-lastError case.
Comment thread apps/mobile/src/state/use-thread-composer-state.ts Outdated
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 30, 2026
Carry the mobile synchronization repair from pingdotgg#4378 orchestrator-v2-background-waiting while preserving the work owned by pingdotgg#4547 orphaned-output-after-early-settle.
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 30, 2026
…ntime status

The CTM merge left two pingdotgg#4415 sidebar-waiting-state assertions that cannot hold
on this line, because CTM keys thread status off runtime where main uses
session:

- apps/mobile threadListV2.test.ts asserted Waiting through session.idle.
  EnvironmentThreadShell has no session field here, so it did not typecheck.
  pingdotgg#4378 orchestrator-v2-background-waiting already covers the same behaviour
  through the presentation bridge, so the duplicate goes.
- apps/web Sidebar.logic.test.ts asserted that an idle runtime carrying a
  stale lastError resolves to ready. On this line runtime.idle is Waiting: the
  pingdotgg#4378 bridge parks a completed shell at idle only when the post-settlement
  background roster is nonempty. The sibling completed-status assertion still
  covers the stale-lastError case.
mwolson added a commit to mwolson/t3code that referenced this pull request Jul 30, 2026
…ntime status

The CTM merge left two pingdotgg#4415 sidebar-waiting-state assertions that cannot hold
on this line, because CTM keys thread status off runtime where main uses
session:

- apps/mobile threadListV2.test.ts asserted Waiting through session.idle.
  EnvironmentThreadShell has no session field here, so it did not typecheck.
  pingdotgg#4378 orchestrator-v2-background-waiting already covers the same behaviour
  through the presentation bridge, so the duplicate goes.
- apps/web Sidebar.logic.test.ts asserted that an idle runtime carrying a
  stale lastError resolves to ready. On this line runtime.idle is Waiting: the
  pingdotgg#4378 bridge parks a completed shell at idle only when the post-settlement
  background roster is nonempty. The sibling completed-status assertion still
  covers the stale-lastError case.
Comment thread apps/mobile/src/state/use-thread-composer-state.ts Outdated
@maria-rcks
maria-rcks force-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615b Compare August 3, 2026 16:01
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from 70e4972 to 9238867 Compare August 3, 2026 20:20
Comment thread apps/server/src/orchestration-v2/testkit/OrchestratorScenario.ts Outdated
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 2 times, most recently from c5e53c8 to 922f352 Compare August 4, 2026 13:48
Comment thread packages/effect-codex-app-server/src/replay.ts
Comment thread packages/effect-codex-app-server/src/replay.ts
Comment thread apps/server/src/mcp/OrchestratorMcpService.ts Outdated
Comment thread packages/effect-codex-app-server/src/replay.ts
Comment thread apps/server/src/orchestration-v2/testkit/OrchestratorScenario.ts
Comment thread apps/server/src/orchestration-v2/testkit/fixtures/shared.ts Outdated
Comment thread apps/server/src/orchestration-v2/testkit/OrchestratorScenario.ts
Comment thread apps/server/src/orchestration-v2/testkit/fixtures/shared.ts Outdated
Comment thread apps/server/src/mcp/OrchestratorMcpService.ts
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from 827e306 to 4d96e2a Compare August 4, 2026 23:06

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c2f8710. Configure here.

Comment thread apps/web/src/components/SidebarV2.tsx Outdated
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch from e24b59c to 4213ac5 Compare August 5, 2026 14:35
@mwolson
mwolson force-pushed the feat/orchestrator-v2-background-waiting branch from 15d803c to b2c68ba Compare August 5, 2026 14:59
@juliusmarminge
juliusmarminge merged commit f0e1196 into pingdotgg:t3code/codex-turn-mapping Aug 5, 2026
14 of 15 checks passed
@mwolson
mwolson deleted the feat/orchestrator-v2-background-waiting branch August 5, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants