fix(responses): notice a Codex WebSocket that dies after the response started

docs-site/src/content/docs/reference/codex-retry-diagnostics.md

@@ -48,6 +48,39 @@ that the account pool routed to the main account, the earliest known time anothe
 pool account recovers, whichever comes first (never sooner than one minute).
 Resume the Goal after that time; routing then picks an account from fresh state.
 
+## Triage: `idle timeout waiting for SSE`
+
+Codex ends a stream that has sent no event for its `stream_idle_timeout_ms`
+(300 seconds by default) and retries the turn. The report alone does not say
+why the stream went quiet. On the native ChatGPT upstream WebSocket, one
+possible cause is a path that failed silently after the response started: a VPN
+or tunnel restart, a Wi-Fi rejoin or an expired NAT mapping can drop it without
+a FIN or RST reaching the proxy.
+
+On the canonical ChatGPT Codex endpoint
+(`https://chatgpt.com/backend-api/codex/responses`, the exchange with the
+metadata channel), the proxy pings the upstream for the whole exchange. Once the
+response has started, a peer that has answered at least one ping on that
+exchange and then delivers no frame and no pong for two ping intervals ends the
+stream with a retryable failure. Codex then reconnects 30 to 45 seconds after
+the last frame or pong instead of after 300 seconds, and the reconnect row names
+the cause:
+
+```text
+Reconnecting... 1/5 (stream disconnected before completion: Upstream stream terminated unexpectedly: codex websocket stopped answering after the response started [cause=after-response-started ...])
+```
+
+When a ping reaches a path that a restarted tunnel resets, the same row reports
+`codex websocket closed before a Responses terminal event (close 1006)`
+instead, within about one ping interval. An `idle timeout waiting for SSE` that
+still arrives after the full timeout comes from a peer that kept answering pings
+while sending no events, a peer that never answered a ping, or an exchange
+without the metadata channel, such as the OpenAI API Responses WebSocket, which
+sends no pings. All three keep the client's idle deadline on purpose.
+`ocx logs get <request-id> --json` shows the attempt's `codexWsStage` with its
+`pings` and `pongs` counts, which tell the three apart. Raising
+`stream_idle_timeout_ms` only makes them wait longer.
+
 ## Why the first reconnect row can be absent
 
 The public Codex `rust-v0.158.0-alpha.2.1` source gates the ordinary stream-retry

scripts/test-layout/layout.json

@@ -1953,7 +1953,7 @@
     "ws-capacity-reclamation.test.ts": "responses",
     "ws-capacity-tool-ownership.test.ts": "responses",
     "ws-endpoint.test.ts": "responses",
-    "ws-failure-stage.test.ts": "responses",
+    "ws-failure-stage.test.ts": "responses", "ws-committed-liveness.test.ts": "responses",
     "ws-native-injection.test.ts": "responses",
     "ws-native-result-continuations.test.ts": "responses",
     "ws-native-steering.test.ts": "responses",

src/server/responses/codex-ws-exchange.ts

@@ -8,7 +8,8 @@ import { CodexWsMetadata, type CodexWsQuotaObserver } from "./codex-ws-metadata"
 import { CODEX_RESPONSES_HTTP_URL, type PreparedCodexWsRequest } from "./codex-ws-request";
 import { CodexWsCorrelation } from "./codex-ws-correlation";
 import type { CodexWsSession } from "./codex-ws-session";
-import { UPGRADE_DEADLINE_MS, CODEX_WS_LIVENESS_PING_INTERVAL_MS, CODEX_WS_RESPONSE_PRELUDE_TIMEOUT_MS, MAX_CODEX_WS_FRAME_BYTES,
+import { UPGRADE_DEADLINE_MS, CODEX_WS_LIVENESS_PING_INTERVAL_MS, CODEX_WS_RESPONSE_PRELUDE_TIMEOUT_MS,
+  CODEX_WS_COMMITTED_SILENT_INTERVALS, MAX_CODEX_WS_FRAME_BYTES,
   MAX_CODEX_WS_QUEUE_BYTES, markCodexWsResponse, markCodexWsPreludeProjection, normalizeResponsesWsRelayEvent, closedBeforeTerminalMessage,
   codexWsCreateFrameExceedsLimit, codexWsFailureDetail, codexWsPreResponseFailure, markCodexWsStage, codexWsOcxVersion,
   markCodexWsSocketDeath, codexWsWrapperStatus, isCodexWsTerminalEvent, type CodexWsFailureStage, type CodexWsStageRecord,
@@ -155,8 +156,11 @@ export function codexWsExchange(options: ExchangeOptions): Promise<Response> {
     // Liveness while waiting for the first response event (metadata path only): the
     // silence timer is re-armed by every inbound frame or pong; the pinger runs on a fixed
     // interval so a peer that answers pings can never trip the silence bound while alive.
+    // After the commit, each ping tick checks whether any frame or pong arrived since the last.
     let silenceTimer: ReturnType<typeof setTimeout> | undefined;
     let pingTimer: ReturnType<typeof setTimeout> | undefined;
+    let inboundAtLastTick = 0;
+    let silentIntervals = 0;
     // The resolved 200, retained so a later body failure can replace its
     // success-shaped stage record with the failure-shaped one (#4191).
     let committedResponse: Response | null = null;
@@ -260,6 +264,7 @@ export function codexWsExchange(options: ExchangeOptions): Promise<Response> {
       markCodexWsStage(response, stageRecord(null));
       committedResponse = response;
       resolve(response);
+      startCommittedLiveness();
     };
 
     /**
@@ -377,18 +382,40 @@ export function codexWsExchange(options: ExchangeOptions): Promise<Response> {
         CODEX_WS_RESPONSE_PRELUDE_TIMEOUT_MS,
       );
     };
-    /** Ping on a fixed interval until the response starts; a socket without ping() is never pinged. */
+    const pingable = () => typeof (ws as WebSocket & { ping?: unknown }).ping === "function";
+    /** False when ping() threw. It may also close the socket and settle the exchange. */
+    const sendPing = (): boolean => {
+      if (!pingable()) return false;
+      try { (ws as WebSocket & { ping: (data?: string) => void }).ping(); } catch { return false; }
+      pings += 1;
+      return true;
+    };
+    /** Ping on a fixed interval for the life of the exchange; a socket without ping() is never pinged. */
     const schedulePing = () => {
-      const ping = (ws as WebSocket & { ping?: (data?: string) => void }).ping;
-      if (typeof ping !== "function" || responseCommitted || terminal) return;
+      if (!pingable() || terminal) return;
       pingTimer = setTimeout(() => {
-        if (responseCommitted || terminal) return;
-        try { ping.call(ws); } catch { return; }
-        pings += 1;
-        // ping() may close the socket synchronously and settle the exchange; re-check.
+        if (terminal) return;
+        if (responseCommitted && pongs > 0) {
+          const inbound = upstreamFrames + pongs;
+          silentIntervals = inbound === inboundAtLastTick ? silentIntervals + 1 : 0;
+          inboundAtLastTick = inbound;
+          if (silentIntervals >= CODEX_WS_COMMITTED_SILENT_INTERVALS) {
+            failStream(`codex websocket stopped answering after the response started${codexWsFailureDetail(failureStage())}`);
+            return;
+          }
+        }
+        sendPing();
         if (!terminal) schedulePing();
       }, CODEX_WS_LIVENESS_PING_INTERVAL_MS);
     };
+    /** Only a peer that has ponged is judged by its silence, so probe again if none has yet. */
+    const startCommittedLiveness = () => {
+      if (!metadata || terminal || !pingable()) return;
+      inboundAtLastTick = upstreamFrames + pongs;
+      silentIntervals = 0;
+      if (pongs === 0) sendPing();
+      if (!terminal) schedulePing();
+    };
     const onPong = () => {
       if (terminal) return;
       pongs += 1;
@@ -559,7 +586,10 @@ export function codexWsExchange(options: ExchangeOptions): Promise<Response> {
       if (!metadata) commitResponse();
       else if (!responseCommitted && !terminal) {
         armSilence();
-        schedulePing();
+        // Probe now, not one interval in: the path can die seconds after the send, before a
+        // held commit, and the committed check needs a pong from before that.
+        sendPing();
+        if (!terminal) schedulePing();
       }
     };
 

src/server/responses/codex-ws-wire.ts

@@ -25,6 +25,11 @@ export const UPGRADE_DEADLINE_MS = 10_000;
 // keeps exactly the previous 90 s bound; a peer that does can never trip it while alive.
 export const CODEX_WS_LIVENESS_PING_INTERVAL_MS = 15_000;
 export const CODEX_WS_RESPONSE_PRELUDE_TIMEOUT_MS = 90_000;
+// The pinger keeps running after the commit: a tunnel restart or Wi-Fi rejoin can drop the path
+// without a FIN or RST, and codex-rs would wait out its 300 s idle timeout. A peer that has ponged
+// and then sends no frame and no pong for this many intervals is gone, and the stream fails so the
+// client retries. One missed pong is not a verdict.
+export const CODEX_WS_COMMITTED_SILENT_INTERVALS = 2;
 // Keep the push-based WS transport inside the same memory envelope as the
 // bounded SSE relays that consume this response. Unlike fetch response bodies,
 // a WebSocket cannot be paused when a ReadableStream applies backpressure, so
@@ -404,7 +409,7 @@ export type CodexWsFailureStage = {
   firstResponseMs: number | null;
   /** Milliseconds from send to this failure; null when the failure predates the send. */
   elapsedMs: number | null;
-  /** Liveness pings the exchange sent while waiting for the first response event. */
+  /** Liveness pings the exchange sent, before and after the first response event. */
   pings: number;
   /** Pongs the peer answered with; zero on a peer that never answers pings. */
   pongs: number;

structure/transports/responses-wire-shapes.md

@@ -480,7 +480,7 @@ A standalone no-response
 exchange has a 90-second prelude deadline in addition to the upgrade deadline.
 That prelude deadline is a ceiling, not a floor: the exchange runs under the
 caller's abort signal, so a `connectTimeoutMs` shorter than 90 seconds cancels
-an already-sent create before the prelude timer fires.
+an already-sent create before the prelude timer fires. After the first Responses event an exchange with the metadata channel keeps pinging on the same interval, and sends one ping at the commit unless an earlier one was answered; a peer that has answered a ping on that exchange and then delivers no frame and no pong for two consecutive intervals is treated as a dead socket, so the committed stream fails as a body error that the relay ends with its `upstream_reset` `response.failed` tail. This check never cuts a peer that keeps answering, however long the model is silent; that peer, like one that never answered a ping, still meets the client's own idle deadline (`tests/responses/ws-committed-liveness.test.ts`).
 These are transport-fidelity guarantees, not a provider-billing guarantee.
 
 Every exchange also leaves a content-free stage record (`CodexWsStageRecord`, #4191): create-frame bytes (measured on failure only — the committed-success record keeps it null so the happy path never byte-counts a megabyte replay frame), send completion, numeric close code, elapsed, first-frame and first-response-event durations, frame counters, liveness ping/pong counts, pool reuse, the OCX/Bun versions, and (#6740) `terminalEvent` — the normalized `response.completed` / `response.failed` / `response.incomplete` / `error` frame that ended a plain exchange, observed before any synthetic tail — with `terminalErrorStatus`, the wrapper status of a terminal `error` frame. Both are null when no terminal frame arrived, on native-control exchanges (which can end on an acknowledgement), and on rows written before they existed; malformed persisted values read as null without dropping the stage. The exchange pins the record on the resolved Response (`markCodexWsStage`, the same marker seam as `markCodexWsResponse`); `handleResponses` adopts it onto the serving attempt, and usage.jsonl persists it per attempt behind a drop-guard normalizer, so hand-edited rows cannot inject strings into the DTO. Later snapshots update the same response-local record in place, so an attempt holding the committed reference observes final success or failure counters. Each exchange supplies a complete fresh snapshot; separate responses keep distinct records. On eager-relay cancel-drain expiry, upstream cancellation finalizes the transport snapshot before the cancellation hook writes the usage row; an actual terminal observed within the drain still wins over cancellation. The record never carries conversation text, headers, close-reason text, or account identifiers, and it is not a fallback-eligibility signal: nothing it says permits a resend. The one replacement an operator can grant after a socket dies is the resend gate's decision (see [ambiguous-resend gate](responses-failover.md#ambiguous-resend-gate)).

tests/fixtures/test-layout-expected.json

@@ -1340,7 +1340,7 @@
   "ws-capacity-reclamation.test.ts": "responses",
   "ws-capacity-tool-ownership.test.ts": "responses",
   "ws-endpoint.test.ts": "responses",
-  "ws-failure-stage.test.ts": "responses",
+  "ws-failure-stage.test.ts": "responses", "ws-committed-liveness.test.ts": "responses",
   "ws-native-injection.test.ts": "responses",
   "ws-native-result-continuations.test.ts": "responses",
   "ws-native-steering.test.ts": "responses",