[Feature]: add durable stream-stage timeline and failure attribution to request history #1217

Area

Streaming / request history / observability

Current status

OpenCodex already has several pieces of correlation-safe request observability:

• every request has an OpenCodex requestId;
• request history is durably backed by usage.jsonl and exposed through the paginated /api/request-history API;
• firstOutputMs records semantic TTFT;
• failure rows can persist fields such as errorCode, terminalStatus, closeReason, and redacted upstreamError;
• durable routeDecision traces explain why a provider/model/account route was selected;
• the live request-log context can classify some terminals using transportPhase and terminalSource.

However, the durable request-history record still cannot reconstruct the end-to-end streaming timeline.

In particular, transportPhase and terminalSource are currently live request-log diagnostics rather than durable usage.jsonl fields, and they do not identify the actual failure side precisely enough.

This issue tracks the remaining durable streaming-stage observability.

Problem

When a Codex turn remains on “thinking” or ends with a stream-disconnected error, an operator should be able to determine from one request-history record whether the delay or failure occurred:

• before OpenCodex dispatched upstream;
• while waiting for upstream response headers;
• while reading the upstream response body/SSE stream;
• while transforming or relaying upstream data;
• while writing downstream;
• because the client cancelled;
• during terminal-event delivery.

Today several of these cases can collapse into the same terminal status.

For example:

terminalSource = synthetic
transportPhase = mid_stream
status = 502

does not prove that the provider caused the failure.

A synthetic terminal identifies who manufactured the terminal event, while a broad relay catch may cover both upstream reads and local relay/rewriting work.

Required durable timeline

Add a bounded, payload-free stream timeline to the canonical request-history row.

At minimum record elapsed milliseconds from request acceptance for:

upstreamDispatchMs
upstreamHeadersMs
upstreamFirstByteMs
upstreamFirstSemanticOutputMs
downstreamFirstWriteMs
upstreamEndMs
downstreamEndMs

Existing firstOutputMs may satisfy upstreamFirstSemanticOutputMs if the naming/relationship is explicit and backwards compatible.

All timings are best-effort observability only.

Failure to record a timing must never affect request delivery.

Failure attribution

Add stable bounded enums that identify where the terminal condition originated.

For example:

failureSide:
  upstream
  relay
  downstream
  client
  local

failureStage:
  pre_dispatch
  upstream_wait_headers
  upstream_read
  relay_transform
  downstream_write
  client_cancel
  terminal_delivery

Exact enum names may follow repository conventions, but the contract must distinguish these cases without parsing free-form error text.

transportPhase and terminalSource may remain for compatibility, but they must not be treated as substitutes for causal attribution.

Interpretation

The resulting record should support deterministic conclusions such as:

no upstreamDispatchMs
  -> failure occurred before upstream dispatch

upstreamDispatchMs but no upstreamHeadersMs
  -> provider/network failure before response headers

upstreamFirstByteMs but no downstreamFirstWriteMs
  -> OpenCodex relay/transform path

downstreamFirstWriteMs then upstream_read failure
  -> upstream/network mid-stream failure

upstreamEndMs but no downstreamEndMs
  -> downstream relay/client path

Example

{
  "requestId": "ocx-...",
  "status": 502,
  "durationMs": 61342,
  "firstOutputMs": 9107,
  "streamTimeline": {
    "upstreamDispatchMs": 12,
    "upstreamHeadersMs": 4410,
    "upstreamFirstByteMs": 4421,
    "downstreamFirstWriteMs": 4423,
    "upstreamEndMs": 61340,
    "downstreamEndMs": 61342
  },
  "failureSide": "upstream",
  "failureStage": "upstream_read",
  "transportPhase": "mid_stream",
  "terminalSource": "synthetic"
}

Persistence

The timeline and attribution must be part of the canonical durable request-history representation, not only the in-memory /api/logs ring.

They must therefore survive:

ocx stop
ocx start

and remain available from:

GET /api/request-history/:requestId

The request-history SQLite index remains a derived projection; usage.jsonl stays canonical unless the underlying storage contract is intentionally changed separately.

Protocol coverage

Cover both:

• native Responses passthrough;
• translated/provider-adapted streaming paths.

Instrumentation should be defined at shared transport/relay boundaries where possible rather than independently inventing incompatible timelines per adapter.

Non-streaming requests may omit stream-only fields.

Request correlation

The existing OpenCodex requestId remains the primary correlation key.

Where protocol-compatible and not already present, return it to clients as:

x-opencodex-request-id

so a client-visible failure can be matched to the durable request-history record without exposing internal state.

Privacy and bounds

The feature must not persist:

• authorization headers or credentials;
• prompts or request bodies;
• images;
• raw SSE payloads;
• account identifiers that are not already part of the approved history contract;
• resolved peer addresses;
• unbounded exception text;
• arbitrary stack traces.

Timeline fields are numeric timestamps/durations and failure attribution uses closed enums.

Existing redaction and bounded-metadata rules continue to apply.

Relationship to existing work

Already delivered and not part of this issue:

• durable request-history indexing and pagination;
• firstOutputMs TTFT;
• durable route-decision traces;
• existing terminal status/error diagnostics;
• existing live transportPhase / terminalSource classification.

Related issue #919 established why terminalSource="synthetic" is insufficient for causal attribution: the synthetic event producer is not necessarily the underlying failure source.

#820 remains the broader architecture/performance tracker. This issue owns only correlation-safe streaming-stage observability.

Acceptance criteria

Durable timeline

• Canonical request history records upstream dispatch time.
• Canonical request history records upstream response-header time.
• Canonical request history records first upstream body/SSE receipt.
• Existing semantic TTFT is explicitly represented.
• Canonical request history records first successful downstream write.
• Canonical request history records upstream body end/failure time.
• Canonical request history records downstream end/cancel/failure time.
• Timeline fields survive proxy restart.

Attribution

• Stable bounded failure-side classification exists.
• Stable bounded failure-stage classification exists.
• Upstream header wait is distinguishable from upstream body-read failure.
• Upstream body-read failure is distinguishable from relay/transform failure.
• Relay/transform failure is distinguishable from downstream write failure.
• Client cancellation is distinguishable from upstream failure.
• Terminal-delivery failure is distinguishable from the underlying stream failure.
• Attribution does not depend on parsing arbitrary exception text.

Coverage

• Native Responses passthrough is covered.
• Translated provider streaming is covered.
• Eager and non-eager relay paths produce compatible stage semantics.
• Requests with no semantic model delta remain diagnosable.
• Non-streaming requests remain backwards compatible.

Safety

• Instrumentation is best-effort and cannot fail request delivery.
• No raw request or SSE payload is persisted.
• No credential material is persisted.
• No new unbounded text fields are introduced.
• Added metadata remains within existing request-history size/bounding expectations.

API

• GET /api/request-history/:requestId exposes the durable timeline and attribution.
• Existing request-history rows without the new fields remain readable.
• Existing pagination/index rebuild behavior remains compatible.
• The OpenCodex request ID can be correlated back from the client response where protocol-compatible.