Repository navigation
[Bug]: send-budget-refused retries and failovers leave no attribution in request logs #5044
Description
Client or integration
Any Responses request using adapter retry or account failover.
Area
Proxy observability and retry accounting.
Summary
When the request send budget refuses a retry or account-rotation hop, the runtime returns the already received upstream response but records neither a recovery kind nor a distinct refusal reason. A log with one physical send is therefore ambiguous: it can mean no recovery was eligible, or recovery was eligible but withheld by the send budget.
This was isolated while closing #4856. It did not cause that reporter's Antigravity failure, which was fixed by #4913, but it made the initial evidence impossible to interpret.
Confirmed source path
createAdapterPhysicalSendthrowsSendBudgetExhaustedErrorwhenreserveDispatchrefuses.fetchGoogleWithRetrycatches that error whilependingResponseexists and returns the original response without recording a recovery kind.- The account-rotation loop breaks when a hop is refused, again without recording that rotation was considered and withheld.
- Persisted request logs therefore show the same
sendCountand emptyrecoveryKindsas a request that never attempted recovery.
Expected behaviour
Record a bounded, non-secret reason when recovery is withheld, separately from physical sends. The log should distinguish at least:
- retry not eligible;
- retry eligible but send budget refused;
- account rotation eligible but send budget refused; and
- recovery dispatched.
The field must not contain credentials, account ids, raw upstream bodies, prompt content or arbitrary exception text. Existing send counts must remain physical-send counts rather than being inflated by refused attempts.
Suggested regression
Exercise an adapter 429 retry and an account-rotation path with an injected exhausted budget. Assert one physical send, the original response preserved, no fabricated recovery send, and a stable refusal attribution in the persisted/request-log projection.
Relationship to #4856
#4856 is resolved and should stay closed. This is the separate observability gap identified by its final source analysis, not another quota or rotation patch.
Activity
lidge-jun commentedon Sep 18, 2026
리뷰 · 우선순위 68 / 80
이 이슈는 요청 send budget이 재시도나 계정 로테이션 hop을 거부했을 때, 런타임은 이미 받은 upstream 응답을 돌려주지만, 요청 로그에는 “recovery를 고려했다가 budget 때문에 참았다”는 기록이 안 남는다는 관측성 구멍입니다. #4856을 닫는 과정에서 분리됐고, 그 리포터의 Antigravity 실패 자체는 #4913으로 고쳤습니다. #4856은 닫힌 채로 두고, 이 이슈만 따로 다루는 게 맞습니다.
지금
dev에서 관련 경로는 이렇습니다.createAdapterPhysicalSend/reserveDispatch가 거절하면SendBudgetExhaustedError를 던집니다.src/adapters/google-http.ts의fetchGoogleWithRetry는 그 에러를 잡고pendingResponse가 있으면 원 응답을 반환합니다. 이때recovery를 로그에 push하지 않습니다. 계정 로테이션 루프도 hop이 거절되면 그냥 break합니다. 그래서 persisted request log의sendCount와 빈recoveryKinds만 보면, “처음부터 recovery 자격도 없었다”와 “자격은 있었는데 send budget이 막았다”를 구분할 수 없습니다. #4856 조사 때 증거가 읽히지 않았던 이유가 바로 이것입니다.기대는 비밀·계정 id·원문 body·프롬프트·임의 exception 텍스트 없이, bounded한 거부 사유를 physical send count와 분리해 남기는 것입니다. 최소 네 갈래: (1) retry 비자격, (2) retry 자격인데 send budget 거부, (3) account rotation 자격인데 send budget 거부, (4) recovery 실제 dispatch. sendCount는 거부 시도로 부풀리면 안 됩니다. 제안 회귀(주입 exhausted budget으로 adapter 429 retry + rotation 경로)도 타당합니다.
라인 114-116 (src/adapters/google-http.ts) -
SendBudgetExhaustedError+pendingResponse면 원 응답 반환, recoveryKinds 기록 없음.경로 src/adapters/physical-send.ts -
reserveDispatch거절 시SendBudgetExhaustedError. 거절 자체는 맞지만 상위의 attribution sink가 없다.경로 src/server/request-log.ts (
recoveryKinds) - 스키마는 이미 배열로 존재. “budget-refused” 같은 안정 토큰을 넣을 자리가 있다.경로 #4856 / #4913 - 원인 버그는 닫혔고, 이 이슈는 그 조사의 관측성 후속이다. 다시 quota/rotation 패치를 열 일이 아니다.
메인테이너의 판단이 필요한 지점
recoveryKinds에 넣을지, 별도recoveryWithheldReasons필드를 둘지. 기존 필드를 재쓰면 대시보드/쿼리가 “recovery가 일어났다”로 오해할 수 있다.너의 추천
인정하되 관측성-only 좁은 PR로 받는다. physical send count는 그대로 두고, budget 거부 시 안정적인 non-secret reason만 기록. Google retry + account-rotation 두 fixture로 회귀. #4856은 재오픈하지 말 것.
이 댓글은 grok-bot이 작성했습니다
feat(usage): attribute a recovery the send budget withheld (#5056)
lidge-jun commentedon Sep 18, 2026
Fixed on
devby #5056.recoveryWithheldis a bounded per-attempt vocabulary recorded besiderecoveryKinds. The four states you asked to distinguish fall out of the pair rather than needing four values: neither field set means nothing was eligible; a withheld reason means something was and the budget refused it; a recovery kind means one was dispatched.sendCountdoes not move for a refusal, which was your explicit constraint and is also why the adapter seam isonRecoveryWithheld— a sibling of the existingonPhysicalSendrather than another field on it, so an adapter cannot report a refusal as a send even by accident.Recorded at the four places that fell through silently: the Google retry ladder while it holds a retryable response, and the generic OAuth rotation in the run-turn, passthrough and adapter-continuation loops. The continuation one distinguishes "the budget refused" from "rotation found no next account", which produce the same response today.
The suggested regression is covered as you framed it. A 429 and a 503 with a prepaid budget assert one physical send, the original response preserved by identity, no fabricated recovery send, and a stable attribution. A 404 asserts nothing withheld, which is what keeps the empty case meaningful — a field that is always set says nothing. The persisted projection is pinned separately: recognized reasons round trip, a repeated reason deduplicates, an unrecognized one is dropped rather than failing the row, and an attempt that withheld nothing omits the key so old rows keep their exact shape.
The vocabulary is
retry-send-budgetandrotation-send-budget, validated on read, so a malformed log line cannot introduce arbitrary text into the projection.Ingwannu commentedon Sep 18, 2026
Reopening after verifying current
devatc5b39d43e6.#5056 records the Google adapter refusal and several main-response rotation paths, but the original observability contract is still incomplete. The generic OAuth loop in
src/server/responses/adapter-dispatch.tsstill breaks on a refusedreserveCredentialHopwithoutnoteAttemptRecoveryWithheld; the generic and Anthropic sidecar loops insidecar-execution.tsreturnnullon the same refusal without recording it. Same-target guards that skip a retry on an exhausted budget also remain indistinguishable from an ineligible retry.recoveryWithheldis persisted in raw attempts, but the Dashboard log type/reason rendering does not expose it, so the operator-facing log still cannot answer why recovery did not run.Required closure: instrument every budget-refusal exit through the shared bounded enum, add generic OAuth and both sidecar regressions, and render the closed reasons in the log UI without exposing credentials or arbitrary errors. Physical send counts must remain unchanged.