Skip to content

fix(openai): rebuild native WS failover from bounded current-window history - #8770

Open
Hato626 wants to merge 1 commit into
Wei-Shaw:mainfrom
Hato626:fix/native-ws-current-window-replay
Open

Hato626 wants to merge 1 commit into
Wei-Shaw:mainfrom
Hato626:fix/native-ws-current-window-replay

Conversation

@Hato626

@Hato626 Hato626 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Add bounded, connection-local current-window history for native Responses
ctx_pool WebSockets, so an eligible later-turn account failure can rebuild the
current request instead of requiring a reconnect for every later-turn
account failure. It never falls back to replaying the connection's first frame.

Based on upstream main at 530c5e3a4 (v0.2.21). This is a standalone patch:
no fork-only BPS implementation, frontend changes, database migrations, production
configuration, credentials or production infrastructure are included.

Existing HTTP bridge support vs. this change

HTTP-bridge current-turn reconstruction is already in the upstream code.
82cbe6aff7 added the later-turn recovery path; current main already retains
wsAccountFailoverReplayInput, calls buildOpenAIWSCurrentTurnRetryPayload from
the bridge loop, and carries the rebuilt request in
newOpenAIWSCurrentTurnFailoverError. This PR does not reimplement that feature.

#8564 correctly made native WS fail closed after the first turn because its
existing replay state was not a complete conversation and replaying the original
first frame was unsafe. This PR preserves that safety boundary when complete
current history is unavailable, and supplies the missing native history when it
can be safely reconstructed. A small shared-handler change refreshes routing,
billing model and image capability from the current retry request; this also
benefits the existing bridge retry path. The bridge transport itself is unchanged.

Behavior

  • Retain structured client input and successful terminal response.output,
    including assistant messages, reasoning/encrypted content and native tool calls.
    Terminal output order is authoritative, even if streamed item events differ.
  • Extend history once per turn; share immutable old bodies, detach new input
    bodies once, and avoid duplicating a complete prefix resent by the client.
  • A new root after compaction replaces the old window. On an already-eligible
    failure before downstream-visible output, rebuild the current input, remove the
    old account's previous_response_id, and retain the current client model and
    unscoped identity source for the next account's mapping/scoping.
  • Consecutive account failures reuse the rebuilt current request, not the first
    connection frame. Existing retry limits, scheduling and slot behavior remain.
  • Unknown anchors, incomplete responses, unresolved item references, missing
    tool-call pairs, unsupported input shapes or exhausted budgets remain fail
    closed. No tool output is converted into a user message. Healthy forwarding
    continues without the cache; unsafe migration requires reconnect/resubmission.
  • Default per-connection retained-history budget: 64 MiB / 32768 items,
    allocated on demand. Either zero limit disables caching; negatives are invalid.
    Overflow discards the entire cache; a new complete root can restore it.
    There is no aggregate cache admission limit. These limits are not an RSS
    bound or a guarantee against OOM across many large concurrent sessions.
  • History is neither persisted nor shared across disconnected sessions. This
    does not promise exactly-once execution after ambiguous upstream failures or
    universally portable account-specific encrypted state.

Memory and performance

Measured on an isolated Linux amd64 host (Intel Xeon 6982P-C), Go 1.27.3,
GOMAXPROCS=2. Runtime/allocation figures are medians of 5 runs with
-benchtime=1s; heap figures are medians of 3 runs with explicit GC.

Normal-turn replay maintenance

One operation is a 128-turn conversation, with a distinct 1 KiB or 10 KiB
ASCII user input and an equally sized assistant output each turn. The BEFORE
control runs the existing upstream input-replay extraction/merge helpers; AFTER
runs the same work plus native history retention and terminal-output parsing.
These are component comparisons in the same benchmark binary, not a full
before/after gateway throughput benchmark. Wire fixtures are prepared outside
the measured loop. Networking, inference and unrelated ingress work are excluded.

Input + output per turn Time BEFORE -> AFTER (128 turns) Extra time per turn Cumulative allocation BEFORE -> AFTER New retained-history accounting/session
1 KiB + 1 KiB 8.188 -> 8.766 ms (+7.1%) 4.51 us 0.428 -> 1.930 MiB 0 -> 0.276 MiB
10 KiB + 10 KiB 71.705 -> 74.205 ms (+3.5%) 19.54 us 0.521 -> 4.399 MiB 0 -> 2.526 MiB

B/op is total allocated bytes over 128 turns, not retained heap or peak
RSS. The last column covers only the new cache (JSON bytes + 64 bytes/item),
not pre-existing replay storage. This is an intentional memory/CPU cost for a
new recovery capability, not a claim of lower memory usage or end-to-end speedup.
Previously owned history bodies are shared, rather than deep-copied each turn.

Retained memory BEFORE / AFTER

The opt-in test retains 64 independent 4 MiB root inputs in both cases to
represent existing replay retention, then enables/disables the additional native
cache. This isolates the cache cost; it is not 64 complete gateway processes or
an observation of production RSS after a restart.

GC-live Go heap Existing replay only (BEFORE control) Existing replay + native cache (AFTER)
Before creating sessions 3.79 MiB 3.79 MiB
While all 64 sessions remain live 260.26 MiB 516.77 MiB
After close and GC 3.76 MiB 3.76 MiB

The extra live heap is approximately 256.508 MiB total / 4.008 MiB per session
for this workload. Cache accounting is 268441344 bytes and returns to zero on
close. All 64 sessions are admitted, demonstrating that there is no shared cache
quota. A separate boundary test retains 62916140 bytes (just over 60 MiB), then
releases the whole cache when the next 4 MiB delta exceeds 64 MiB; a fresh root
successfully resumes caching. The maximum is not preallocated per connection.

Account-switch reconstruction

Serializing a retained 4 MiB full context into a retry request costs a median
11.597 ms, 23.791 MiB cumulative allocations, and 24 allocations.
This is measured only when reconstruction is invoked, not on every healthy
turn. There is no equivalent successful native later-turn recovery to benchmark
on the old fail-closed path, so its BEFORE runtime is N/A, not zero.
The complete retry path also has existing defensive payload copies that this
serialization microbenchmark does not include. Transient memory, allocator
rounding, the existing replay state and process RSS remain outside the cache cap.

Reproduce

cd backend
GOMAXPROCS=2 go test ./internal/service -run '^$' \
  -bench '^BenchmarkOpenAIWSNativeHistory' -benchmem -benchtime=1s -count=5
SUB2_HISTORY_MEMORY_TEST=1 GOMAXPROCS=2 go test ./internal/service \
  -run 'TestOpenAIWSNativeHistory(MemoryPressure|LargeSession)$' -v -count=3
GOMAXPROCS=2 go test -race ./internal/service ./internal/handler \
  -run 'TestOpenAIWSNativeHistory|TestOpenAIResponsesWebSocket_Native' -count=3

These are synthetic measurements, not live OpenAI latency/quota results or a
real 1M-token conversation. Concurrent large sessions and simultaneous rebuilds
can still exhaust process memory; operators must size concurrency accordingly.

Tests

  • Full internal/service, internal/handler and internal/config suites pass
    both on the untouched upstream baseline and on this PR. No failing test was
    excluded to obtain a green result.
  • Focused native history/WS handler tests pass under Linux -race, three runs.
  • Full golangci-lint v2.14.0 run ./... passes with 0 issues, using the
    upstream CI configuration. git diff --check also passes.
  • Both benchmark cases (five runs), memory controls and 64 MiB overflow/recovery
    tests (three runs) pass. The opt-in memory tests are not enabled in normal CI.
  • Service coverage: ownership, full-prefix deduplication, compaction/root reset,
    repeated rebuild, incomplete/unknown chains, exact byte limits, output overflow,
    item limits, disabled cache, concurrent accounting and cleanup.
  • Gateway-handler integration uses real local WebSockets and three mock accounts:
    same-window continuation, window 67 -> 68, two consecutive account failures,
    model changes, reordered item events, exhausted budgets, unknown anchors and
    no transparent replay after visible output.
  • Opt-in memory tests cover 64 independent 4 MiB sessions, with/without native
    history, plus a session growing past 60 MiB and releasing history at 64 MiB.

Separate live verification of the implementation before this clean upstream
port also used two authorized
ChatGPT OAuth accounts: text, function/custom tools and real remote compaction
continued across accounts in both tested directions. Only the pre-output quota
failure was injected to trigger switching. Those credential-dependent scripts
are intentionally not part of this PR; they do not establish universal provider
compatibility or constitute a real 1M-token load test.

Copilot AI balanced review requested due to automatic review settings November 13, 2026 21:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants