Skip to content

fix(gateway): stream interval watchdog ignores upstream heartbeat lines - #7956

Open
xcrong wants to merge 1 commit into
Wei-Shaw:mainfrom
xcrong:fix/stream-watchdog-ping-blindness-7955
Open

xcrong wants to merge 1 commit into
Wei-Shaw:mainfrom
xcrong:fix/stream-watchdog-ping-blindness-7955

Conversation

@xcrong

@xcrong xcrong commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7955.

Problem

Anthropic long streams emit periodic heartbeats (: ping comments, event: ping, data: {"type":"ping"}). The SSE scanner goroutines refreshed lastReadAt on every line read, so a stalled upstream that only sends heartbeats kept gateway.stream_data_interval_timeout (default 180s) fed forever. Observed in production: two /v1/messages (claude-opus) streams went silent after the first event and hung for 16 minutes with zero data events, until downstream timeouts tore them down. No failover triggered because there was no error status - just silence.

Change

  • New helper anthropicSSELineIsHeartbeat() (gateway_service.go): classifies heartbeat lines (empty, : comments, event: ping, ping-typed data payloads). Everything else (including [DONE] and unrecognized lines) conservatively counts as data.
  • The three Anthropic streaming pumps (OAuth main path, APIKey passthrough, native passthrough) and anthropicNativeLinePump.next() no longer reset the data-interval timer on heartbeat lines.
  • Error strings (stream data interval timeout), partial-usage billing, and the no-failover-after-write rule on the Claude path are all unchanged - this PR only makes the watchdog fire on time (fail fast), it does not add mid-stream account switching.

Tests

  • TestAnthropicSSELineIsHeartbeat: table test for the classifier.
  • TestGatewayService_StreamingIntervalTimeoutFiresOnPingOnly, TestAnthropicPassthrough_IntervalTimeoutFiresOnPingOnly, TestAnthropicNativeLinePump_PingOnlyStillTimesOut: ping-only hung upstreams now trip the watchdog in ~1s (interval=1s in tests).
  • Discrimination verified: the two pump-level tests fail on unpatched code (hang past the test guard) and pass with the fix.
  • Existing suite: go build, go vet, and all Streaming|Stream|Pump|Anthropic|Heartbeat|Failover tests in internal/service pass; gofmt clean.

Out of scope (noted in #7955)

Same per-line refresh pattern exists in bedrock/openai/antigravity/images pumps; happy to follow up separately if maintainers want it generalized.

Anthropic long streams emit periodic heartbeats (: ping comments,
event: ping, data {"type":"ping"}). The SSE scanner goroutines
refreshed lastReadAt on every line read, so a stalled upstream that
only sends heartbeats kept gateway.stream_data_interval_timeout fed
forever - observed as 16-minute silent hangs with zero data events.

Count only real data lines toward the data-interval watchdog in the
three Anthropic streaming pumps (OAuth main path, APIKey passthrough,
native passthrough) and in anthropicNativeLinePump. Error strings and
failover semantics are unchanged; a ping-only stream now fails fast
with 'stream data interval timeout' plus partial usage billing.

Fixes Wei-Shaw#7955
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA. ✅
Posted by the CLA Assistant Lite bot.

@xcrong

xcrong commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

@xcrong

xcrong commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

recheck

github-actions Bot added a commit that referenced this pull request Oct 9, 2026
riofutab added a commit to riofutab/sub2api that referenced this pull request Oct 9, 2026
- 合入上游 Wei-Shaw#7956:Anthropic 流式转发的数据间隔看门狗不再被 ping 与注释行刷新,上游只发心跳时按时超时

测试:随整合分支统一验证(unit 全量、golangci-lint、前端检查)。
riofutab added a commit to riofutab/sub2api that referenced this pull request Oct 9, 2026
- 合入上游已合并的 Wei-Shaw#7618:平台改为数据驱动,接入 Command Code 与 Cline,Go 升级到 1.27.2,golangci-lint 升级到 v2.14.0。
- 合入开放 PR Wei-Shaw#7920 Wei-Shaw#7924 Wei-Shaw#7928 Wei-Shaw#7930 Wei-Shaw#7931 Wei-Shaw#7933 Wei-Shaw#7935 Wei-Shaw#7936 Wei-Shaw#7939 Wei-Shaw#7941 Wei-Shaw#7947 Wei-Shaw#7951 Wei-Shaw#7954 Wei-Shaw#7956 Wei-Shaw#7960。
- 新增 Decisions 决策接口(Wei-Shaw#7922)及其与 TypeSafe System One 的双向互转(Wei-Shaw#7948),上游 404 时切换账号。
- Haiku 5.5 的 top_p 违规请求在网关直接拒绝,OAuth 伪装不再在 top_p 旁补 temperature,工具集 beta 过滤覆盖 Haiku 5.5。
- 修正新增代码的静态检查告警。

全量 unit 测试、golangci-lint、前端全量 vitest 均已通过;集成测试本机无 Docker,待 CI 验证。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] StreamDataIntervalTimeout 被上游心跳行持续刷新,静默断流可挂住十几分钟

1 participant