feat(server): resume unavailable provider sessions with mobile-safe status - #1
feat(server): resume unavailable provider sessions with mobile-safe status#1rix1337 wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
cd34688 to
0ee71d9
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
rix1337
changed the title
0ee71d9 to
0ded6cb
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
0ded6cb to
8f2351b
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
8f2351b to
235784d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 235784dbca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| type: "thread.usage-limit-resume.retry", | ||
| commandId: yield* providerCommandId(event, "usage-limit-resume-retry"), | ||
| threadId: thread.id, | ||
| resumeAt, | ||
| attempt: usageLimitResume.attempt, |
There was a problem hiding this comment.
When a rejected initial prompt has a pendingMessageId and its first automatic attempt fails through an adapter-emitted runtime.error, this retry command omits that ID. The resulting scheduled event replaces usageLimitResume, so the following attempt sends the generic continuation instead of replaying the original task and attachments. Fresh evidence beyond the earlier fix is that this runtime-ingestion path does not copy usageLimitResume.pendingMessageId, unlike the reactor and startup-recovery retry paths.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 842e2ef. Runtime-error retries now retain pendingMessageId, preserving the saved task reference instead of silently switching to a generic continuation. Focused ingestion coverage checks that the ID survives the retry and subsequent failed completion.
| <SettingsSection id="automatic-resume" title="Automatic resume"> | ||
| <SettingsRow | ||
| {...searchableSetting("automatic-resume")} | ||
| description="Automatically continue after provider limits or temporary outages. Turning this off cancels pending retries in this environment. Each retry sends a continuation prompt." | ||
| control={ | ||
| <Switch | ||
| checked={settings.enableAutomaticResume} |
There was a problem hiding this comment.
When the current hosted web client connects remotely to a pre-feature server, enableAutomaticResume decodes to its default true, so this section displays an enabled switch even though that server cannot honor the setting or schedule retries; toggling it merely sends a setting the old environment does not support. The newly published threadUsageLimitResume capability is otherwise unused, so gate both this row and its search entry on that capability as other version-skewed settings are gated.
AGENTS.md reference: AGENTS.md:L74-L74
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 842e2ef. Both the settings section and its search result require threadUsageLimitResume === true from the connected primary server. The supported/unsupported capability cases are tested.
235784d to
842e2ef
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 842e2ef2f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| payload: { | ||
| threadId: command.threadId, | ||
| resumeAt: current.nextAttemptAt, | ||
| attempt: current.attempt, | ||
| updatedAt: thread.updatedAt, |
There was a problem hiding this comment.
When two repair paths race for the same in-flight attempt—for example, sendTurn rejects while the provider event stream also emits a typed runtime.error—the second retry reaches this stale branch after the first has installed the next schedule. The re-emitted payload omits current.pendingMessageId, and the projector replaces the entire resume object, so the next attempt loses the rejected user's prompt and attachments and sends the generic continuation instead. Preserve the current pending message ID in this no-op re-emission.
Useful? React with 👍 / 👎.
| const USAGE_LIMIT_MESSAGE = | ||
| /(?:usage|rate) limit|spend cap|model (?:is )?(?:currently )?at capacity|(?:provider|service|server) (?:is )?temporarily unavailable|quota (?:has been )?(?:exceeded|reached|exhausted)|too many requests|\b429\b|resource[_ ]exhausted|insufficient[_ ]quota/i; |
There was a problem hiding this comment.
The bare \b429\b alternative also matches source locations such as Adapter.ts:429:12. Non-request turn-start failures are classified from Cause.pretty(...), which includes stack traces, so an unrelated authentication, validation, or process failure can be mistaken for a usage limit solely because one frame lands on line 429, causing automatic retries for a permanent error. Require HTTP/status context here; structured numeric status fields are already handled separately.
Useful? React with 👍 / 👎.
1 participant
What Changed
Port PR pingdotgg#8577 to v0.0.38 with default-on automatic continuation after usage limits, spend caps, capacity errors, and provider 404 outages. A server setting provides opt-out and cancels pending retries.
Provider reset times, including dated AM/PM messages, take priority with a one-minute cushion. Otherwise waits are 3 × 20 minutes, 5 × one hour, then six hours. Timers recheck the wall-clock deadline before dispatch because production logs showed early wakeups being rejected and stranding schedules. Restart recovery restores pending work without reviving cancelled schedules.
Existing mobile message events carry one updated, clearly labeled T3 system notice with the next attempt. New resume event tags are withheld from legacy mobile/unknown clients. Repeated resumable failures do not append error activities. No new mobile status enum or dependency is introduced.
Why
Unfinished work should not require a person to repeatedly check provider availability. Every retry sends a real continuation prompt in the existing conversation, so the increasing waits limit unnecessary traffic. The server owns scheduling because clients disconnect and the desktop app restarts.
This is a release-specific continuation of an existing feature proposal, not an unrelated rewrite. Current-message reset metadata now takes precedence over empty or stale Codex account-limit data. Initial ACP prompt rejections schedule retries even without a runtime error event.
The PR remains large and does not meet the preferred small-fix size. Most additions implement the original cross-provider feature and its regression coverage. Ponytail review removed 83 lines of unused client opt-in helpers; cancellation, durability, and compatibility checks remain necessary. This PR targets the user's fork, not a claim of upstream acceptance.
UI Changes
Settings → Integrations has a default-on Resume when available switch. One conversation notice replaces stale duplicate status banners and shows the next attempt in server time. Existing Working awareness is used only while scheduled.
Before/after screenshots and interaction video are not yet available. Actual App Store mobile UI verification remains outstanding; unmodified v0.0.38 schema and routing checks passed but do not prove every released client.
Validation
311 focused tests passed before the final clock fixes; the updated parser/reactor set passes 77 tests. Server/web typechecks and the desktop build passed for the preceding revision; final deployment checks are recorded separately. Earlier patch variants ran on multiple production clients, but successful post-limit continuation is not yet established in production. Logs proved one continuation reached Codex and hit its usage limit again.
Implemented with Codex in T3 Code. Exact model identifier is unavailable in this session.
Checklist