Skip to content

[Bug report / diagnostics]: Native Responses pre-output latency persists in 2.76/2.77 A/B; need supported stage attribution and recovery guidance #6568

Closed as not planned

Description

@kirin-bot139

Client or integration

Codex CLI

Area

Proxy and routing

Summary

Native Codex requests on Windows show materially longer pre-output waits while output delivery after the first visible delta is broadly unchanged. The installation is usable, but latency and intermittent pre-header connection failures remain unresolved. Expected: prompt request delivery, clear stage attribution, and a supported recovery path for the observed failures.

This is a field report asking for supported diagnosis, not a claim that OpenCodex is uniquely responsible. Existing observations do not isolate provider queueing, prefill, network upload/handshake, and all proxy/client behavior. Separate issue #6567 covers mixed historical tok/s formulas; correcting that display must not count as repairing this latency.

Reproduction

  1. Use the official Codex CLI through the installed native OpenCodex Responses route and existing ChatGPT login.
  2. Run codex exec --ephemeral --skip-git-repo-check --sandbox read-only --json --color never -m gpt-6.1-sol -c model_reasoning_effort=\"medium\" - from an empty directory.
  3. Provide a harmless fixed prompt instructing it not to use tools and to repeat The quick brown fox jumps over the lazy dog. exactly 12 times, one line each.
  4. Correlate completion events with the local request log. Each compared request had one physical send, no replay, the same model/effort and default service tier.
  5. Compare a short controlled 2.77 → 2.76 → 2.77 sequence with production configuration and selected account unchanged. The retained 2.77 installation was restored after the trial.

Observed first visible output / request duration, milliseconds:

Phase Sample 1 Sample 2 Sample 3
2.77 before trial 14682 / 18068 14028 / 17524 13604 / 17071
2.76 trial 14288 / 18125 5735 / 8188 6958 / 10755
2.77 restored 14169 / 17848 4887 / 7405 10362 / 14096

Input was approximately 17.3k tokens; output 135–143 tokens including 9–17 reasoning tokens. Cache hits varied. The small sequential sample cannot establish or exclude a version-level latency cause; it shows that reverting versions did not consistently remove the wait.

Version

OpenCodex 2.77.0, controlled comparison with 2.76.0. Bun 1.4.0. Official Codex CLI 0.160.0.

Operating system

Windows 11 x64; exact Windows feature-update version was not included in this report.

Provider and model

Native OpenAI Codex Responses, GPT-6.1 Sol, medium reasoning, default service tier. Same existing account selection policy and serving account across the correlated test requests. Outbound HTTP proxy in use; no VPN node or account was switched for the version comparison.

Logs or error output

A temporary loopback observer forwarded only these diagnostic requests/responses unchanged to the existing OpenCodex service and retained scalar timing, not authentication, request bodies or response text. Two samples with the same 83,938-byte client request yielded:

Observation, ms since local ingress 2.76 2.77
Response headers 2644 3311
response.created received 2647 3312
First output item 12620 14434
First delta 13095 15049
Last delta 16920 18841
response.completed received 17156 18842
Response stream end 17203 18844
Upstream completed_at - created_at 14000 16000

Upstream timestamps have whole-second precision. Their difference corroborates a substantial response lifetime but does not identify the backend phase. Completed WS attempt records have requestBytes: null, limiting upload-stage attribution.

Real failures include:

Provider connect timeout after 200000ms
upstream_reset_replay_refused
codex websocket closed before a Responses terminal event (no-upstream-frame)

A later 15:17 UTC+08:00 observation, already posted in the comments, also contains repeated upstream Our servers are currently overloaded. Please try again later. responses. This does not establish a service-wide outage or prove that all slow requests share one cause.

Screenshots and supporting files

Only redacted scalar results are included. Raw logs, account identifiers, private prompts, request/conversation identifiers and business data are not attached.

Broader observational windows on 2026-10-04: 09:00–10:00 local time had 905 successful medium requests, median first visible output 5435.5 ms. Later natural windows were slower, including about 20 seconds following a machine restart. Consistent visible-token/window estimates remained around 33–34 tok/s. Same-conversation examples with fewer input tokens were also slower, so context growth alone is not established as the explanation. These windows are not identical-input experiments.

The native WS pool/session/exchange/request files compared unchanged between official 2.76 and 2.77 archives. Twelve offline same-input request-build tests produced equal destinations, headers, bodies and WS frames; this is not coverage of the entire live pipeline.

Two local compatibility changes were retained and disclosed: known-native unsupported legacy minimal effort normalization, and use of the existing 250 ms bounded reader for optional Windows catalog advice. Ordinary medium requests are not downgraded; unknown/stale advice remains suppressed. The bounded advice change reduces a known local advisory wait but does not eliminate this latency.

Redacted configuration

Native Responses route; provider request pacing off; no forced-off upstream WebSocket override; official default 200000 ms connect deadline. The client-facing WebSocket setting was not treated as the upstream setting.

A temporary forced-HTTP override previously caused a distinct slowdown and was removed. Normal successful requests again have real upstream WS and connection-reuse evidence. No global Fast, model/effort downgrade, account rotation, retry-safety override, unknown-result replay or private credential extraction was used.

Requested guidance:

  1. Which supported diagnostic separates upload/upgrade, upstream accepted-to-generation, and local relay delay without capturing user content or credentials?
  2. Is a known compatibility or routing regression consistent with these measurements? Please identify a supported mitigation or next minimal falsifiable test rather than treating HTTP 200 as latency acceptance.
  3. Can completed WS attempts preserve the actual sent-byte count and bounded stage timing instead of requestBytes: null?
  4. If attribution lies outside OpenCodex, what upstream escalation and minimal private diagnostic fields are supported?

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. github-actions commented on Oct 4, 2026

    @github-actions
    Contributor

    Issue reopened

    The report now contains the information required by the automated check. Thanks for updating it.

  2. kirin-bot139 commented on Oct 4, 2026

    @kirin-bot139
    ContributorAuthor

    Additional live observation, 2026-10-04 15:17 UTC+08:00, with production configuration unchanged during this observation:

    • Latest 10-minute request-start window: 134 already-finalized records, 125 HTTP 200 and 9 HTTP 502; still-running requests are excluded.
    • Successful medium-effort requests: n=109, median first visible output 12,980 ms, median request duration 18,135 ms. The same visible-token/window estimate used for the morning baseline was 34.50 tok/s versus 33.90 tok/s in the morning.
    • Median time outside the recorded upstream attempts was 257 ms (95th percentile 391 ms). All 109 successful medium samples have upstream WS attempt evidence.
    • Fresh failures now repeatedly carry the upstream message Our servers are currently overloaded. Please try again later., alongside other upstream-processing errors and some 200000 ms pre-header timeouts. This is observed error text, not an inference solely from latency. The installed TypeScript sources contain no locally-authored match for that overload sentence.

    These receipts support a real upstream-failure component but do not establish a service-wide outage or prove that every slow request has the same cause. No pacing, global Fast, model/effort, account selection, VPN, or retry-safety setting was changed to obtain this window. No unknown-result operation was replayed. Request identifiers remain private.

  3. added
    bugSomething isn't working
    proxyHTTP proxy, routing, reverse-proxy / management auth
    on Oct 4, 2026
  4. Ingwannu commented on Oct 4, 2026

    @Ingwannu
    Owner

    Keeping the latency report separate from #6567. In your observer pair, response.created to first delta is 10.448 s on 2.76 and 11.737 s on 2.77; there is also 2.644/3.311 s before response headers. Those are observed intervals, not provider queue/prefill measurements. The A/B overlap and overload receipts do not prove a single cause, a service-wide outage, or that the version rollback fixes the wait.

    Current source explains the successful WS byte-count gap: codex-ws-exchange.ts intentionally records requestBytes: null on committed success to avoid a UTF-8 scan of potentially multi-megabyte replay frames; failures measure the actual serialized frame. This is an explicit instrumentation/performance tradeoff, not evidence that a frame was absent. The inbound observer's 83,938 bytes cannot substitute for the serialized WS create-frame size.

    For the next attribution check, use already-retained, redacted scalar records if available: transport/reuse, sent flag, frame/event counts, firstFrameMs, firstResponseMs and elapsedMs, alongside the observed first visible output. Correlate locally and keep identifiers/content/credentials private. Those stage fields can narrow observed transport/relay intervals but do not attest upload completion or provider-internal queue/prefill. A successful-frame byte count or new phase counters needs a bounded-cost instrumentation change and offline regression proof; it must not be presented as a latency fix.

    Do not enable payload/body usage-debug capture just for this scalar question, force a transport/effort/account change, or override unknown-delivery replay safety. Preserve existing failure receipts for upstream support rather than rerunning an unknown-result operation. I inspected source only: no native CLI, live account/provider request, runtime change or security/privacy scan was run. This remains unresolved, not a shipped mitigation.

  5. lidge-jun commented on Oct 7, 2026

    @lidge-jun
    Owner

    Thank you for the careful timing comparisons. They do not isolate an OpenCodex regression, and the retained observations include explicit upstream overload errors. Please reopen with current OpenCodex/client versions, sanitized ocx doctor output, and a minimal comparison that identifies the additional delay within the proxy without capturing private content.

    Closing as not reproducible with the current evidence; this is not a fix claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingproxyHTTP proxy, routing, reverse-proxy / management auth

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions