Skip to content

[Bug] 2.79.0: opted-in main credits go stale after 5 min and requests get a local 429 with a weekly retry-after #6692

Closed
Closed
[Bug] 2.79.0: opted-in main credits go stale after 5 min and requests get a local 429 with a weekly retry-after#6692
Labels
account-poolOAuth, credentials, Codex pool, quota, failover, plansbugSomething isn't working

Description

@sungyongcho
sungyongcho
opened on Oct 6, 2026

Client or integration

Codex App

Area

Authentication and account pool

Summary

With the main account opted in under "Use credits" and holding a positive ChatGPT credit balance, requests that arrive after the included weekly window is full fail intermittently with a local 429 rate_limit_exceeded (CodexMainAccountCreditsOffError). The refusal is local: it takes about 50 ms and nothing is sent upstream. Codex then reports exceeded retry limit, last status: 429 Too Many Requests and drops the turn. Requests a few seconds later succeed again.

The cause is the credit freshness clock introduced in 6ef255f (#6466 carry, v2.77.0):

  • hasSpendableCodexCredits() rejects credit evidence older than CODEX_CREDITS_FRESHNESS_MS (5 min), and mainCreditsHoldResetAt() then refuses the main account.
  • The credit balance is re-observed only by a WHAM read. As the commit comment says, ordinary usage headers do not re-observe it.
  • Nothing on the request path refreshes stale main credits. primeCodexPoolQuotas() → primeMain returns early whenever a main quota exists and the reason is not priority-failback.
  • In practice, the dashboard's /api/provider-quotas poll (60 s, pauseWhenHidden) is what re-observes main credits, through fetchMainAccountInfoSnapshot() with MAIN_CACHE_TTL = 5 min. The refresh cadence is therefore at least 5 min plus poll alignment, and it stops while the dashboard is hidden. Every cycle leaves a window where the evidence is older than 5 min, and every main request in that window is refused.
  • The refusal's retry-after is the weekly reset (about 275,000 s here), so Codex stops retrying instead of waiting the seconds a refresh would take.

Restarting the proxy "fixes" it for about 5 minutes, because startup priming observes credits once.

The pool path calls the same predicate (isCodexAccountHeldForCredits → hasSpendableCodexCredits), so opted-in pool accounts are likely exposed in the same way. I observed it only on the main account.

This is not #6658. That trace is a physical Anthropic 429 with upstream_reset_replay_refused; this one is a local refusal with no upstream send. The symptom in Codex is the same.

Related: #6571 / #6572 (credit-consent verdict, fixed), #6000 (usage-limit error shape).

Reproduction

  1. Exhaust the main account's included weekly window while it has a positive ChatGPT credit balance.
  2. Allow the main account under "Use credits" (creditCodexAccountIds contains __main__).
  3. Route Codex through the proxy and send a request: it is served from credits.
  4. Leave the dashboard hidden, or simply wait until more than 5 minutes have passed since the last credit observation (codex-quota-cache.json → quotas.__main__.credits.observedAt).
  5. Send another request: it gets a local 429 rate_limit_exceeded in about 50 ms with retry-after set to the weekly reset. Codex shows exceeded retry limit, last status: 429 Too Many Requests.
  6. Open the dashboard, or restart the proxy, so credits are re-observed: the next request succeeds.

Version

2.79.0 (installed .deb); code path unchanged on dev at 9099e4b (2.80.0)

Operating system

Ubuntu (Linux 7.0, GNOME), desktop app (opencodex-desktop) owning the runtime

Provider and model

openai (native Codex main account) / gpt-6-astra and gpt-6.1-sol

Logs or error output

# Main credit observations (codex-quota-cache.json observedAt), 2026-10-06 local time:
20:02:25  (proxy start)
20:06:37  (+4m12s)
20:12:33  (+5m56s)  <- evidence was stale from 20:11:37 to 20:12:33

# usage.jsonl, the request inside that window (identifiers removed):
{"timestamp":"20:11:44","provider":"openai","model":"gpt-6-astra","status":429,"durationMs":64,
 "errorCode":"rate_limit_exceeded","failureStage":"headers-only","failureCause":"rate-limit",
 "upstreamError":"Codex main account reached its usage limit, and spending ChatGPT credits is off or no fresh spendable balance is available. ..."}

# Codex log for the same kind of refusal:
POST http://127.0.0.1:10100/v1/responses status=429 Too Many Requests ... "retry-after": "275000"
Turn error: exceeded retry limit, last status: 429 Too Many Requests

# Same day: 287 served GPT requests, 7 refused this way, all with the same message.

Screenshots and supporting files

No response

Redacted configuration

{
  "accountPoolStrategy": "quota",
  "creditCodexAccountIds": ["__main__", "<pool-1>", "<pool-2>"],
  "codexQuotaAutoRefresh": { "__main__": { "weekly": true } }
}

Main quota at the time: weeklyPercent: 100, credits: { hasCredits: true, overageLimitReached: false, allowed: true, balance: >0 }.

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. github-actions
    added
    bugSomething isn't working
    account-poolOAuth, credentials, Codex pool, quota, failover, plans
    on Oct 6, 2026

  2. MadKangYu commented on Oct 7, 2026

    @MadKangYu
    MadKangYu
    on Oct 7, 2026

    Reproduced on macOS (launchd service, Codex Desktop + codex exec 0.160.1, opencodex 2.79.0 from npm).

    Measured on 2026-10-07 (local time, KST):

    • __main__ added to creditCodexAccountIds at 09:29. Weekly window at 100%, credit balance about 56,970.
    • 09:32: one request was served from credits (right after a manual POST /api/codex-auth/accounts/refresh).
    • 09:43 to 12:52: 36 of 36 OpenAI requests were refused locally (429, 3 to 12 ms, failureStage: headers-only). No dashboard was open.

    A second case with the same shape, which #6693 may not cover:

    • Before the reset, the stored main quota said weeklyPercent: 100 and weeklyResetAt 2026-10-10 06:17.
    • The weekly window actually reopened earlier. After a manual refresh at 12:53, the stored quota became weeklyPercent: 0 and weeklyResetAt 2026-10-14 12:51.
    • Requests at 12:51 and 12:52 were still refused locally, because the stored reading was still 100%.
    • Refused requests produce no response headers, so the reading cannot correct itself through traffic. I did not confirm whether the recovery sweep would have re-read it before the stored 10-10 reset.

    Question: does the recovery-sweep refresh in #6693 also re-read main usage while the stored weekly window is full, and not only the credit balance? If so, it covers both cases.

  3. Ingwannu commented on Oct 7, 2026

    @Ingwannu
    Ingwannu
    on Oct 7, 2026
    Owner

    @MadKangYu On #6693 head d24100d, yes, a successful admitted sweep reads the whole WHAM usage response, not just credits: fetchMainAccountInfoWhileOwned parses usage and main-policy quota, remembers credits, and publishes the current-credential quota through setAccountQuotaFromParsed. For the case you describe (stored usage full, positive opted-in credits, evidence old enough), that can update a newly reopened weekly window as well.

    This is conditional, not a fix guarantee. The new worker skips missing/empty/retracted/refused credits, blocked hard lock, needsReauth, active query backoff and unavailable native-main ownership; it is not a general refresh of every held main account. #6693 is also not landed: the owner's expired-bearer P1 must be resolved, and exact-head runtime CI is action_required. Please include your stale-100% => fresh-0% case in its synthetic recovery regression, asserting the actual quota-store update and subsequent admission without a dashboard/manual refresh, alongside the expired-access-token/valid-refresh-grant control. No additional paid request or production refresh is needed to build that fixture.

  4. WOULDU-pres commented on Oct 7, 2026

    @WOULDU-pres
    WOULDU-pres
    on Oct 7, 2026
    Contributor

    Reproduced on macOS (2.79.0, npm global install, CLI only, dashboard not open), with one addition about pool accounts.

    Main account: same as the report

    • Weekly window at 100%. Main is opted into credits and has a positive balance (allowed: true, overageLimitReached: false).
    • The proxy restarted at 02:38:26 (auto-update 2.78.0 → 2.79.0). Requests were served from credits until 02:43:41, which is the 5-minute freshness window after the startup prime.
    • The next requests (04:39–04:41) were refused locally in 4–51 ms with CodexMainAccountCreditsOffError and an empty attemptSequence.
    • ocx account refresh openai re-observed the credits, and the next request (04:41:50) was served from credits again.

    Pool accounts look exposed too, and neither #6693 nor #6669 covers them

    I read this from dev at c30b522 and haven't reproduced it live, because my only pool account currently needs reauth.

    • isCodexAccountHeldForCredits → hasSpendableCodexCredits gates a pool account at two points:
      • at selection, in isCodexAccountHeldAtUsageLimit (src/codex/routing/selection.ts:91-126), which takes it out of rotation;
      • at dispatch, in assertPoolAccountCredits (src/codex/auth-context.ts:1447, plus the dispatch guard at :757).
    • Once the evidence goes stale, nothing on that path observes it again:
      • The recovery sweep only claims accounts whose cooldown source is reset-derived (src/codex/routing/probe-lease.ts:106). A locally held account never gets one.
      • The lazy prime in resolveCodexAuthContext runs after the credit assert at :1447, and only when the account has no quota at all or priority failback is on.
    • Header-driven quota updates set a new updatedAt but carry the previous credits forward with its original observedAt (src/codex/quota.ts:660). That's correct for spending authority, but any freshness or cache check keyed on updatedAt will treat an actively serving account as fresh while its credit evidence ages out.
    • In a multi-account pool, I'd expect the visible effect to be traffic quietly rotating away from the credited account rather than a 429. A 429 should show up only when that account is pinned, or when every account is held.

    A question: the x-codex-credits-* response and WS metadata is passed through to clients (src/server/safe-response-headers.ts:4) but never recorded as an observation. Is that deliberate? If it's trustworthy enough, recording it would keep the evidence fresh for actively serving accounts, main and pool alike, without extra WHAM reads. That still leaves the idle or held case to a sweep like #6693's.

  5. sungyongcho commented on Oct 7, 2026

    @sungyongcho
    sungyongcho
    on Oct 7, 2026
    ContributorAuthor

    @lidge-jun @Ingwannu Could you share your view on the intended refresh and recovery behavior across main and pool accounts?
    Following the reproductions here and #6707, I would like to clarify how the current fixes fit together:

    These seem related through the lifecycle of the information used to admit an account. We need to refresh evidence that can restore a usable account, while avoiding repeated automatic attempts against a grant already known to require reauthentication.

    Would it make sense to agree on a few shared expectations for main and pool?

    1. Credit freshness and recovery should not depend on keeping the dashboard open or on the held account receiving traffic.
    2. Expired credit evidence, an empty or restricted balance, a temporary query failure, and a revoked refresh grant should have distinct recovery behavior.
    3. Background reads should respect credential ownership, prepare valid credentials where permitted, and retain the existing generation checks, deduplication and backoff.
    4. A refusal caused by stale evidence should be distinguishable from exhausted quota, without weakening spending consent or promising that a retry will succeed.

    For #6693 specifically, the expired-bearer concern remains unresolved. I would appreciate guidance on whether to continue that implementation or consolidate the main-account fix around #6669.

    Would you prefer to keep the remaining pool freshness/recovery coverage in this issue, or close this as a main-account fix and track that part separately? I am happy to keep the work here scoped to the agreed behavior and regression cases.

  6. lidge-jun
    added 2 commits that reference this issue on Oct 7, 2026

    fix(codex): renew opted-in main credits before expiry (lidge-jun#6692)

    cda6402

    fix(codex): renew opted-in main credits before expiry (#6692) (#6669)

    bbd4e33

  7. lidge-jun commented on Oct 7, 2026

    @lidge-jun
    lidge-jun
    on Oct 7, 2026
    Owner

    Fixed on dev by #6669 (bbd4e33): opted-in main credit evidence is now renewed by the background sweep from 3 minutes of age (tokens prepared first, passive probe that never quarantines), so admission no longer falls back to a local 429 when the 5-minute freshness window lapses. It ships with the next release cut from dev.

  8. lidge-jun
    closed this as completedon Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    account-poolOAuth, credentials, Codex pool, quota, failover, plansbugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions