Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add bounded, connection-local current-window history for native Responses
ctx_poolWebSockets, so an eligible later-turn account failure can rebuild thecurrent request instead of requiring a reconnect for every later-turn
account failure. It never falls back to replaying the connection's first frame.
Based on upstream
mainat530c5e3a4(v0.2.21). This is a standalone patch:no fork-only BPS implementation, frontend changes, database migrations, production
configuration, credentials or production infrastructure are included.
Existing HTTP bridge support vs. this change
HTTP-bridge current-turn reconstruction is already in the upstream code.
82cbe6aff7added the later-turn recovery path; currentmainalready retainswsAccountFailoverReplayInput, callsbuildOpenAIWSCurrentTurnRetryPayloadfromthe bridge loop, and carries the rebuilt request in
newOpenAIWSCurrentTurnFailoverError. This PR does not reimplement that feature.#8564 correctly made native WS fail closed after the first turn because its
existing replay state was not a complete conversation and replaying the original
first frame was unsafe. This PR preserves that safety boundary when complete
current history is unavailable, and supplies the missing native history when it
can be safely reconstructed. A small shared-handler change refreshes routing,
billing model and image capability from the current retry request; this also
benefits the existing bridge retry path. The bridge transport itself is unchanged.
Behavior
response.output,including assistant messages, reasoning/encrypted content and native tool calls.
Terminal output order is authoritative, even if streamed item events differ.
bodies once, and avoid duplicating a complete prefix resent by the client.
failure before downstream-visible output, rebuild the current input, remove the
old account's
previous_response_id, and retain the current client model andunscoped identity source for the next account's mapping/scoping.
connection frame. Existing retry limits, scheduling and slot behavior remain.
tool-call pairs, unsupported input shapes or exhausted budgets remain fail
closed. No tool output is converted into a user message. Healthy forwarding
continues without the cache; unsafe migration requires reconnect/resubmission.
allocated on demand. Either zero limit disables caching; negatives are invalid.
Overflow discards the entire cache; a new complete root can restore it.
There is no aggregate cache admission limit. These limits are not an RSS
bound or a guarantee against OOM across many large concurrent sessions.
does not promise exactly-once execution after ambiguous upstream failures or
universally portable account-specific encrypted state.
Memory and performance
Measured on an isolated Linux amd64 host (Intel Xeon 6982P-C), Go 1.27.3,
GOMAXPROCS=2. Runtime/allocation figures are medians of 5 runs with-benchtime=1s; heap figures are medians of 3 runs with explicit GC.Normal-turn replay maintenance
One operation is a 128-turn conversation, with a distinct 1 KiB or 10 KiB
ASCII user input and an equally sized assistant output each turn. The BEFORE
control runs the existing upstream input-replay extraction/merge helpers; AFTER
runs the same work plus native history retention and terminal-output parsing.
These are component comparisons in the same benchmark binary, not a full
before/after gateway throughput benchmark. Wire fixtures are prepared outside
the measured loop. Networking, inference and unrelated ingress work are excluded.
B/opis total allocated bytes over 128 turns, not retained heap or peakRSS. The last column covers only the new cache (JSON bytes + 64 bytes/item),
not pre-existing replay storage. This is an intentional memory/CPU cost for a
new recovery capability, not a claim of lower memory usage or end-to-end speedup.
Previously owned history bodies are shared, rather than deep-copied each turn.
Retained memory BEFORE / AFTER
The opt-in test retains 64 independent 4 MiB root inputs in both cases to
represent existing replay retention, then enables/disables the additional native
cache. This isolates the cache cost; it is not 64 complete gateway processes or
an observation of production RSS after a restart.
The extra live heap is approximately 256.508 MiB total / 4.008 MiB per session
for this workload. Cache accounting is 268441344 bytes and returns to zero on
close. All 64 sessions are admitted, demonstrating that there is no shared cache
quota. A separate boundary test retains 62916140 bytes (just over 60 MiB), then
releases the whole cache when the next 4 MiB delta exceeds 64 MiB; a fresh root
successfully resumes caching. The maximum is not preallocated per connection.
Account-switch reconstruction
Serializing a retained 4 MiB full context into a retry request costs a median
11.597 ms, 23.791 MiB cumulative allocations, and 24 allocations.
This is measured only when reconstruction is invoked, not on every healthy
turn. There is no equivalent successful native later-turn recovery to benchmark
on the old fail-closed path, so its BEFORE runtime is N/A, not zero.
The complete retry path also has existing defensive payload copies that this
serialization microbenchmark does not include. Transient memory, allocator
rounding, the existing replay state and process RSS remain outside the cache cap.
Reproduce
These are synthetic measurements, not live OpenAI latency/quota results or a
real 1M-token conversation. Concurrent large sessions and simultaneous rebuilds
can still exhaust process memory; operators must size concurrency accordingly.
Tests
internal/service,internal/handlerandinternal/configsuites passboth on the untouched upstream baseline and on this PR. No failing test was
excluded to obtain a green result.
-race, three runs.golangci-lint v2.14.0 run ./...passes with 0 issues, using theupstream CI configuration.
git diff --checkalso passes.tests (three runs) pass. The opt-in memory tests are not enabled in normal CI.
repeated rebuild, incomplete/unknown chains, exact byte limits, output overflow,
item limits, disabled cache, concurrent accounting and cleanup.
same-window continuation, window 67 -> 68, two consecutive account failures,
model changes, reordered item events, exhausted budgets, unknown anchors and
no transparent replay after visible output.
history, plus a session growing past 60 MiB and releasing history at 64 MiB.
Separate live verification of the implementation before this clean upstream
port also used two authorized
ChatGPT OAuth accounts: text, function/custom tools and real remote compaction
continued across accounts in both tested directions. Only the pre-output quota
failure was injected to trigger switching. Those credential-dependent scripts
are intentionally not part of this PR; they do not establish universal provider
compatibility or constitute a real 1M-token load test.