[chrisohearn]

Ledger key: r2/provider-fallback. Implements the automatic version of #41.

Problem

A run that hit an account limit before doing any work stopped the thread. The fallback model setting only chose a model for new threads, and auto-resume needs a reset time, which ChatGPT sharing caps never report. On 2026-11-12 the Jarvis inbox (run 20) stopped on subscription_sharing_usage_limit_exceeded with resetAt: null. Switching the thread to another ChatGPT account then failed twice (runs 21 and 22) with "This conversation is open in another Codex client": the first account's shared app-server still held the conversation's writer lock (lslocks shows one codex process holding a flock per loaded thread).

What changes

Failover. A failed run moves to the configured fallbackModelSelection and its message is sent once more there when all of these hold:

• The adapter proved at its boundary that the turn did no work (turn.terminal.noWorkObserved) and named an account cause (accountFailure):
  • Codex: a once-set flag on the turn context, set by every output delta, reasoning, plan, item start or completion (except the user message and compaction) and every approval or input request. It is set before the early returns that keep file changes out of the projection, and it spans capacity-retry contexts. Account causes: usage_limit (usageLimitExceeded, rateLimitExceeded, exhausted 429, sharing caps), and auth (sharing or grant codes that reject the account, unauthorized).
  • Claude: a whitelist at the SDK frame handler. Non-work frames are results, rate-limit events, prompt echoes, init, retry, status, compaction and hooks, and assistant frames that carry an API-failure error. Any permission request counts as work. Account causes: 401 or an auth error frame (auth), 402 or billing_error (billing), and rate-limit rejection evidence or a plain 429 (usage_limit).
  • Excluded: Cursor, Grok, OpenCode, Antigravity, Kiro, ACP, Pi and Muse set no flag, so they never fail over.
• The fallback is configured for the project, is on another provider instance, and this is the run's first attempt.
• The run is not itself a failover or restart continuation, and its message is a prompt, not a wake, delegated result, compaction, slash command or workflow script.
• Nobody steered the failed run. A steer would be lost if only the first message were re-sent, so a steered run skips its claim.

Once-only claim. run.failover = { state: "pending", reason, modelSelection } is written in the same writeIfRunCurrent as the failure, with a replay-safe run.failover outbox effect.

• While the claim is pending (and its effect unsettled), nothing treats the run as finished: handleRunUpdate defers queue promotion and delegated-completion finalization, startup queue recovery waits, settle is blocked, and thread waits and parent tasks treat it like a pending restart continuation.
• The effect sends a message.dispatch with failoverOfRunId; each effect attempt is its own command id, so a rejected attempt is retried rather than replayed. A message from an outside client is re-sent within that client's mode limit (senderClient.access). The orchestrator re-checks the claim and that nothing ran since. It also checks there is no Stop request, steer, archive or pending runtime request, that the model matches, and that the text and sender are the source message's own. Clients cannot set failoverOfRunId or failoverNotice; intake strips them. In one commit it then writes a system_notice ("Switched to Claude Opus 5.5 after ChatGPT - Revotech hit its usage limit."), creates the new run on the fallback model (same text, context and attachments; the delegated task carries over), and marks the source dispatched with the new run id.
• Every other exit marks the claim skipped: a guard refusal, Stop, or a last effect attempt that fails or dies. That run.updated runs the deferred reactions exactly once and reports the blocker. If even that skip cannot be written, the claim is resolved as skipped later: by startup recovery, by the next queue start, or by settle, whichever comes first.
• Messages queued for the failed instance before the failover follow the thread to the fallback when they start, so they do not switch back and fail again. A message queued for that instance afterwards keeps its pick.
• The fallback gets the prompt once: the source run is left out of the handoff.
• Duplicate deliveries replay the receipt.

Precedence. Failover wins when it claims a run. Auto-resume candidates and the resume guard ignore runs whose failover is pending or dispatched. A skipped claim falls back to today's limit recovery. MCP events no longer emit thread.blocker for a run that is failing over, so a subscriber like Jarvis does not recover the same failure twice. A skip still reports the blocker.

Instance-switch lock. When thread/resume fails with the in-use error and the latest attempt on that native conversation ran on a different provider instance, turn start now takes the existing fresh-native-session path with the full-history recovery handoff. Before this change it failed the run. The same-instance case (another real client writing) still fails, as before. I chose this over releasing the old session: the lock belongs to the old instance's shared app-server, which serves other threads, and thread/unsubscribe does not release it promptly. Continuity is kept whenever the old instance did unload the thread.

Standing instructions. Full-summary handoffs (provider switches, failover to another driver, queued starts that need full context, restarts, and the fresh-session recovery above) name the thread's standing instructions by item id in T3's own context line: "Item is this thread's first message: a role, rules or limits it sets still apply to you." No item carries a marker, and any item line that starts with [Historical is escaped, so fetched pages or tool output cannot forge a header. The standing item is only the opening run's own message (the first run since the latest fresh-context run), and only when the user or the thread's launcher sent it: never a post, wake, delivery, compaction, workflow script, linked sender or another thread. Selection reserves it first when it fits; items stay intact, so an oversized one is omitted and listed for t3_thread_read. The handoff record stores the id so delivery names the same item. Recovery now applies the fresh-context boundary to its history too. Fork handoffs name no standing item. A linked agent's message never qualifies, item lines that start with [Historical behind blanks or invisible characters are escaped too, and so is the standing sentence when item text repeats it. Recovery for a failover run leaves its source out of history.

No migration: the claim lives in the run payload.

Tests

• Codex terminal classification table: the incident (sharing cap before work, so noWorkObserved and usage_limit); a fileChange start followed by the same failure (no evidence); a sharing sign-in rejection (auth); and a child started via item/completed (no evidence).
• Claude: a 429 with rejection evidence, plain API errors (only 429 is an account cause), an expired login (auth), and assistant text before a 429 (no evidence).
• Stalled-retry rewrite in ProviderSessionManager carries the evidence.
• RunExecutionService: the claim and its effect land in the same write as the failure, and not when work was observed, the cause is missing, or there is no target.
• End to end (ProviderSwitch.integration): the incident through scripted Codex and Claude adapters, the real RunExecutionService, effect worker and orchestrator. The second message fails on subscription_sharing_usage_limit_exceeded before any work, and Claude receives it once with the role named as standing instructions. With no fallback configured the test fails.
• RunExecutionService: a duplicated terminal writes one failure, one claim and one effect.
• Runtime layer (provider failover):
  • The incident shape re-sends once on the fallback with the attachment, delegated task, notice and model switch. The queued message stays queued through startup recovery, then starts on the fallback. Two effect attempts with different command ids create one run, and the handoff names the conversation's first message as standing instructions without repeating the prompt. After a fresh-context run, that is the fresh run's message, and the earlier role never reaches the provider. A failed opening message re-sends with no copy of itself in the handoff.
  • Stop, Stop before the failure landed, archive, a disabled fallback, a steer, a forged text, an outside client's mode limit, and an effect that settled without resolving the claim all skip it.
• Process loss requeues run.failover.
• waitForThread follows a failover run. This fails without the filter change.
• Turn start: a cross-instance in-use resume starts fresh, and the adapter receives the role named as standing instructions. A same-instance in-use resume still fails.
• selectHistory keeps a standing first message that fits ahead of recent work and omits one that does not, naming it only in the context line; a forged item header is escaped.
• Intake strips failoverOfRunId and failoverNotice from client commands.
• MCP blocker suppression table; failover target and claim rules.
• vp run check:changed: format, lint, knip, typecheck (41 packages) and 98 affected test files (2,804 tests) all pass.

Follow-ups

• The handoff wording is model-facing text. No eval covers handoffs today, and I did not add one, so it needs sign-off or an eval case.
• Jarvis's manual relaunch rule from Add launch-only provider fallback with positive no-work evidence #41 should be retired in Jarvis's own instructions; docs/jarvis/upgrades.md already says Jarvis must not relaunch.
• Rolling back to a build without this change leaves a pending run.failover effect undecodable: that build logs a warning on each claim and keeps working, but the row never runs. Claims already dispatched are unaffected.
• Not done: forwarding sourcePlanRef (the source run already completed that plan, so a re-send with it would be refused); a send during the pending window can still start ahead of the failover; a sign-in failure before the session opens no longer claims (only Antigravity reports it, and Antigravity is excluded); hooks count as non-work; the in-use bypass trusts T3's own attempt records.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	❌ Failed 2026-11-12T18:34:11.405992Z We couldn’t complete this review. Comment @codex review to try again.	7204ec1	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[chrisohearn]

Substitute security review (Claude Opus 5.5), done in place of the hosted Codex security review, which is blocked by a usage limit. Head 841f05d. Verdict: merge after fixes. P2-1 and P2-2 block; P2-3 needs an owner decision.

P2 (blocking)

P2-1. The standing-instructions marker can be forged and now carries authority.

• Where: packages/provider-core/src/server/handoffBudget.ts:176, :179-180 and :205.
• For Claude and other non-Codex providers, renderHistory sends the handoff as flat text with no escaping (ContextHandoffDelivery.ts:141). That includes command_execution output.
• Scenario: a fetched page, a repo file or a PR comment contains a [Historical user; user_message; …; standing-instructions] line. On the next failover or switch, Claude sees two "standing" items and is told their rules "still apply to you".
• Fix: identify the standing item by id in the trusted context line (item=<id>), not by a marker inside the item, and neutralise lines that start with [Historical in item text.

P2-2. The label lands on whichever user message comes first, not on the run that firstConversationRunId chose.

• Where: ContextHandoffService.ts:532-535 uses items.find(type === "user_message"). Nothing excludes posts (postOnly, runId: null), wake or notification messages, or non-owner senders.
• Fresh-session recovery (ProviderTurnStartService.ts:1155-1180), which the new cross-instance in-use case now reaches (:1486), gets its items from getTurnStartHistory (ProjectionStore.ts:5170-5184). That has no fresh-context boundary, so a pre-boundary first message the user discarded gets labelled "still applies".
• Restart handoff (Orchestrator.ts:6281) with no fresh-context run ignores the standing run. Fork paths (Orchestrator.ts:8434, ProviderTurnStartService.ts:1296, readForkHistoryItems) apply no boundary.
• Fix: pass the chosen standing item id into prepareProviderHandoff. Accept it only if it is that run's own user message, not a post, wake or notification, and from the user or the launching client. Apply the fresh-context boundary in recovery, or drop the label there.

P2 (owner decision)

P2-3. Failover changes how permissions are enforced.

• Where: CodexAdapterV2.ts:757-763, ClaudeAdapterV2.ts:3066-3073, and ProviderFailover.ts:43-64, which has no mode check.
• Codex auto means a workspaceWrite OS sandbox plus auto-review. Claude auto has no OS sandbox. A Codex auto thread that fails over runs the same prompt without OS containment. Approval requests still gate both.
• Options: skip failover when the target's containment is weaker, require an opt-in, or at least say so in the notice.

P3

1. Claim stuck in pending: a claim stays pending if writing the skip fails (ProviderFailoverDispatch.ts:63-83). thread.blocker then stays suppressed and the deferred queue never runs. Fix: at startup, treat a pending claim whose effect has finished as skipped.
2. Client mode limit not re-checked: the re-send doesn't re-check the outside client's mode limit (ProviderFailoverDispatch.ts:37 vs Orchestrator.ts:15374-15383). Derive it from message.senderClient.access.
3. Server-only failover fields accepted from web clients (contracts orchestrationV2.ts:3329-3331); withCreationProvenance strips only senderClient, and dispatchFailoverMessage (Orchestrator.ts:9278) never checks the text or sender against the source message. Owner-only. Fix: strip the fields at intake, build the notice on the server, and verify the text and sender.
4. Provenance gaps:
   • scheduledTaskId isn't forwarded.
   • Guest attribution is dropped (Orchestrator.ts:15068-15075).
   • mayStartSubagentFollowup (:778-783) grants follow-up rights to any failover.
5. In-use bypass trusts only T3's own records: the cross-instance in-use bypass relies on T3's records (ProviderTurnStartService.ts:757-769, :1486). If an outside Codex client really holds the conversation, T3 now starts fresh, leaving two writers in one worktree. Also require that the old instance's session for this thread is detached.
6. No-work evidence gaps:
   • Hooks count as non-work (ClaudeAdapterV2.ts:4516, Codex hook/completed).
   • Codex mcpServer/elicitation/request (CodexAdapterV2.ts:5855) doesn't set the work flag.
7. Cost: no loops (a failover run can't fail over), but there's no global cap. A transient Codex rateLimitExceeded moves the thread to the fallback for good.
8. Visible text: the notice shows account display names (ProviderFailover.ts:117-128).
9. Cross-account history:
   • It stays within the thread and its fork ancestors.
   • Content has no redaction, so command output goes to the fallback vendor.
   • Projects that already have a fallback set are now opted in (ProjectDefaultsSettings.tsx:254).

Clean

• Runtime mode, interaction mode and worktree are unchanged on re-send.
• Pending approvals, Stop (including one issued before the failure), archive and delete, a mismatched fallback, and a newer run all skip the failover.
• Settle is blocked while a failover is pending.
• Approval requests count as work in both adapters.
• A skip re-emits thread.blocker.
• No tokens are logged.

🤖 Generated with Claude Code

[chrisohearn]

Substitute code review (Claude Opus 5.5), done in place of the hosted Codex code review, which is blocked by a usage limit. Head 841f05d. Verdict: merge after fixes. P2-1 and P2-3 block; P2-2 is strongly recommended.

No P0 or P1 problems.

• Claim and effect: both are written in the same writeIfRunCurrent as the failure, and a duplicate terminal is dropped.
• Restart: the effect is replay-safe, and a second delivery finds the claim dispatched.
• Locks: Stop, dispatch and skip all run under the thread lock, plus the parent lock for child threads.
• No double recovery with auto-resume or restart continuation, and no loops.
• No migration: the claim lives in the run payload.
• Classification: it catches subscription_sharing_usage_limit_exceeded with no reset time. Claude counts only 429 as an account limit and excludes 529.

P2

1. A steer into the failed run is lost.
   • Where: ProviderFailoverDispatch.ts:38-52 re-sends only source.userMessageId. Steers are stored against the run, and they don't count as work.
   • Scenario: during a minute of 429 retries, the user steers "also check X". The fallback gets only the original text.
   • Fix: skip the claim, or include the steers, when any other user message has runId === source.id. Check it in dispatchFailoverMessage.
2. The retry budget is dead.
   • Where: the command id command:provider-failover:${source.id} never changes. A transient dispatchOnce error records a rejection receipt (Orchestrator.ts:15429), and attempts 2-5 just replay it (:15303), so the claim is skipped.
   • Fix: put the attempt number in the command id. The pending-claim check already prevents a double send.
3. The before-start sign-in path reaches only Antigravity.
   • Where: ProviderTurnStartService.ts:886 claims whenever requiresSignIn is set, without checking the provider. Only the ACP/Antigravity adapter sets that flag (AntigravityAdapterV2.ts:275).
   • Effect: an excluded provider fails over, while Codex and Claude never take this path.
   • Fix: gate it to the Codex and Claude drivers.
4. Queued messages undo "the thread stays on the fallback".
   • Where: queued messages keep the failed instance (queueTailModelSelection, Orchestrator.ts:7347), so each one switches back, fails again and fails over again.
   • Fix: when dispatching, move queued runs that are still on the failed instance with no explicit model to the fallback.

P3

1. The fallback sees the prompt twice. isHandoffSourceRun (Orchestrator.ts:1362) includes failed runs, so the handoff carries the source as failed history and then again as the new turn. If it was the first message, it is also marked standing. Leave the source run out of the handoff.
2. The standing item is picked by guessing (ContextHandoffService.ts:532). Pass the chosen item id explicitly; this overlaps with security P2-2.
3. A claim can stay pending forever if skipProviderFailover throws on the last attempt. handleRunUpdate (Orchestrator.ts:15660) never wakes again. Have awaitsFailover's settled-but-unresolved branch emit the skip; this overlaps with security P3-1.
4. A send during the pending window jumps ahead. queueBehindRun (Orchestrator.ts:7328) ignores a pending claim.
5. Fields dropped on re-send: scheduledTaskId and sourcePlanRef.
6. Misplaced doc comment: firstConversationRunId sits between contextBoundaryOrdinal and its doc comment (Orchestrator.ts:1269-1274).
7. Rollback: an older build that the update guard rolls back to can't decode a pending run.failover outbox row.
8. Claude whitelist unconfirmed: if the CLI emits a system/notification frame on a usage-limit rejection, that counts as work. A real rejection transcript would settle it.

Tests

All 7 changed server test files pass: 182, 6 and 741 tests. Gaps:

• The runtimeLayer failover tests write the claim by hand instead of going adapter → RunExecutionService → EffectWorker with retries.
• The skip tests don't assert queue promotion or parent release.
• No duplicate-terminal test.
• No end-to-end test of the incident sequence through the adapter.
• No test that the auto-resume SQL excludes pending claims.
• Missing adapter cases: Claude tool_use-only, partial text, permission request and api_retry; Codex agentMessage start and approval requests.

🤖 Generated with Claude Code

[chrisohearn]

Substitute re-review (Claude Opus 5.5) of the fixes 841f05d..b1ff813 (head ff57d19 is an empty commit; CI is green). Verdict: MERGE. No P0, P1 or P2 problems remain.

Earlier blocking findings, all fixed:

• Sec P2-1, the forgeable marker: the id is named only in the trusted line, and [Historical is escaped.
• Sec P2-2, the wrong standing item: chosen explicitly from the opening run after the latest boundary. Recovery applies the boundary, forks name none, and standingItemId is stored.
• Code P2-1, the steer: guard added. A mutation run confirmed the test catches it.
• Code P2-2, retries: the attempt is now in the command id.
• Code P2-3: the sign-in claim path is removed. Only the Codex and Claude adapters set the no-work evidence.
• Code P2-4, queued runs: queuedBeforeFailoverFrom. A mutation run confirmed the test catches it.
• Pending-forever claims: the startup sweep, queue start and settle all resolve them. The sweep treats pending and running as live, so it can't skip a live claim.
• Sec P3-2 and P3-3 are fixed; Sec P2-3 is closed by the owner's decision.

Upstream-client contract check: no new values were added to existing literal sets. All new contract items are optional keys, and the new literal sets live only inside the optional failover struct, which older decoders drop. No change needed.

Tests run: 230 tests across the changed files, 11 runtime-layer failover cases, and 347 tests in related files, including all of runtimeLayer.test.ts and McpEvents. All pass.

Remaining P3s:

• P3-a Escaping is exact line-start only. A leading space or zero-width character gets through, and an item can copy the trusted sentence. Fix: trim before matching, and escape the trusted sentence in item text.
• P3-b The launcher check doesn't compare sender identity (ContextHandoffService.ts:183). Another outside client, or a linked agent sending through attachments, can qualify as the opener. Fix: reject senderClient.kind === "linked-agent", or match the launcher's sessionId.
• P3-c Recovery (ProviderTurnStartService.ts:1139-1153) drops only run.id. A failover to another Codex instance that hits in-use recovery gets the source run as history, possibly named as standing. Fix: also exclude run.failoverOfRunId.
• P3-d An explicit pick on the failed instance, queued before the failover, moves anyway.
• P3-e A queued slash command on the failed instance switches the thread back to it.
• P3-f A queued fresh-context run keeps the old pending: reservation id. It works; the id is just misleading.
• P3-g The sweep can re-emit a blocker for an older stuck run at startup.
• P3-h Share one "resendable" predicate.
• Test gaps: the no-fallback case fails by timeout rather than an assertion; no test covers reject-then-succeed on retries; slash, post-failover-queue and standing-exclusion cases are missing; no test checks that the blocker is re-emitted.

🤖 Generated with Claude Code