Repository navigation
health-check: add voice-transport probe for Gemini Live close errors - #258
Merged
Merged
Conversation
During the Gemini 3.1 dry-run earlier today, bodhi hit a close code 1011 "exceeded your current quota" within 12 seconds of boot. Every existing health check still said ok (port 9900 listening, watchers registered, bodhi uptime normal, startup banner present) — the probe layer couldn't see the upstream transport failure. I found it only by manually tailing voice-agent.log. Adds check_voice_transport: scans voice-agent.log from the most recent "Sutando — Voice Interface" banner forward, tracks the tail state of the Gemini transport. Reports: - ok "no recent transport errors" when the latest event is "Gemini setup complete" with no abnormal close after it - ok "transport recovered after earlier error" when an abnormal close was followed by a successful setup (auto-recovery worked) - fail "unrecovered transport close: code=N reason=..." when the latest transport event is an abnormal close with no recovery Healthy close codes: 1000 (normal), 4000 (sutando custom goodbye disconnect, bodhi fork commit 44172b8). Everything else is flagged. Catches: - code=1011 Gemini quota exhaustion / tier restriction - code=1007 "Request contains an invalid argument" (CLOSED→CLOSED state machine wedge — the exact pattern I saw earlier today before the bodhi PR #1 fixes landed) - code=1006 abnormal / network drop Wired into run_all_checks right after check_voice_watchers so the voice-agent block in the report is contiguous: voice-agent ok port 9900 voice-watchers ok all 3 watchers active voice-transport ok no recent transport errors Test plan: - ✓ Live check on current clean-boot voice-agent → ok - ✓ 6-case simulated-log harness covering: clean boot, 1011 unrecovered, 1011 recovered, 1000 normal close, 4000 custom goodbye, 1007 wedge unrecovered. All correct. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
sonichi
added a commit
that referenced
this pull request
Apr 10, 2026
Fixes a false positive in the voice-transport probe (#258). Observed live 2026-04-09 at 03:30: voice-agent sat idle for ~10 minutes, then: 03:29:46 [VoiceSession] GoAway from Gemini (timeLeft=50s) 03:30:46 [VoiceSession] Transport closed (state=ACTIVE code=1011 reason="The service is currently unavailable.") 03:30:46 [VoiceSession] Gemini disconnected — will reconnect fresh when client connects That's Google's normal idle-timeout path: Gemini Live fires a GoAway 60s warning, then closes the transport with code=1011 and a "service is currently unavailable" reason. The bodhi session transitions to CLOSED and waits for the next client connect. When a client comes back, `handleClientConnected` reconnects fresh. Not a failure, just the expected lifecycle for an idle voice-agent. The probe was flagging every 1011-after-idle as `fail` with "unrecovered transport close" — which trains users to ignore the signal exactly when we need them to trust it (real 1011s from #259 duplicate-tool or #262 googleSearch misconfig get lost in the noise). Fix: track a `goaway_before_close` flag that flips true on GoAway and false on setup complete. Abnormal closes while the flag is true get reclassified as expected idle-timeouts (not flagged). The flag also clears on setup complete so a RECOVERED idle timeout followed by a fresh unrelated 1011 still correctly fails. Test plan: - ✓ Live probe on current log (has idle-timeout 1011 from 03:30): voice-transport ok - ✓ 4-case simulated-log harness: 1. GoAway → 1011 "service unavailable" unrecovered → ok (was fail) 2. Unrecovered 1011 quota, no GoAway preceding → fail (as before) 3. GoAway → 1011 → setup complete (reconnected) → ok 4. Recovered idle-timeout + fresh 1011 with no new GoAway → fail (proves the flag correctly resets on setup complete) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 tasks done
sonichi
added a commit
that referenced
this pull request
Apr 10, 2026
) Fixes a false positive in the voice-transport probe (#258). Observed live 2026-04-09 at 03:30: voice-agent sat idle for ~10 minutes, then: 03:29:46 [VoiceSession] GoAway from Gemini (timeLeft=50s) 03:30:46 [VoiceSession] Transport closed (state=ACTIVE code=1011 reason="The service is currently unavailable.") 03:30:46 [VoiceSession] Gemini disconnected — will reconnect fresh when client connects That's Google's normal idle-timeout path: Gemini Live fires a GoAway 60s warning, then closes the transport with code=1011 and a "service is currently unavailable" reason. The bodhi session transitions to CLOSED and waits for the next client connect. When a client comes back, `handleClientConnected` reconnects fresh. Not a failure, just the expected lifecycle for an idle voice-agent. The probe was flagging every 1011-after-idle as `fail` with "unrecovered transport close" — which trains users to ignore the signal exactly when we need them to trust it (real 1011s from #259 duplicate-tool or #262 googleSearch misconfig get lost in the noise). Fix: track a `goaway_before_close` flag that flips true on GoAway and false on setup complete. Abnormal closes while the flag is true get reclassified as expected idle-timeouts (not flagged). The flag also clears on setup complete so a RECOVERED idle timeout followed by a fresh unrelated 1011 still correctly fails. Test plan: - ✓ Live probe on current log (has idle-timeout 1011 from 03:30): voice-transport ok - ✓ 4-case simulated-log harness: 1. GoAway → 1011 "service unavailable" unrecovered → ok (was fail) 2. Unrecovered 1011 quota, no GoAway preceding → fail (as before) 3. GoAway → 1011 → setup complete (reconnected) → ok 4. Recovered idle-timeout + fresh 1011 with no new GoAway → fail (proves the flag correctly resets on setup complete) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
liususan091219
pushed a commit
to liususan091219/sutando-fork
that referenced
this pull request
May 28, 2026
…onichi#258) During the Gemini 3.1 dry-run earlier today, bodhi hit a close code 1011 "exceeded your current quota" within 12 seconds of boot. Every existing health check still said ok (port 9900 listening, watchers registered, bodhi uptime normal, startup banner present) — the probe layer couldn't see the upstream transport failure. I found it only by manually tailing voice-agent.log. Adds check_voice_transport: scans voice-agent.log from the most recent "Sutando — Voice Interface" banner forward, tracks the tail state of the Gemini transport. Reports: - ok "no recent transport errors" when the latest event is "Gemini setup complete" with no abnormal close after it - ok "transport recovered after earlier error" when an abnormal close was followed by a successful setup (auto-recovery worked) - fail "unrecovered transport close: code=N reason=..." when the latest transport event is an abnormal close with no recovery Healthy close codes: 1000 (normal), 4000 (sutando custom goodbye disconnect, bodhi fork commit 44172b8). Everything else is flagged. Catches: - code=1011 Gemini quota exhaustion / tier restriction - code=1007 "Request contains an invalid argument" (CLOSED→CLOSED state machine wedge — the exact pattern I saw earlier today before the bodhi PR sonichi#1 fixes landed) - code=1006 abnormal / network drop Wired into run_all_checks right after check_voice_watchers so the voice-agent block in the report is contiguous: voice-agent ok port 9900 voice-watchers ok all 3 watchers active voice-transport ok no recent transport errors Test plan: - ✓ Live check on current clean-boot voice-agent → ok - ✓ 6-case simulated-log harness covering: clean boot, 1011 unrecovered, 1011 recovered, 1000 normal close, 4000 custom goodbye, 1007 wedge unrecovered. All correct. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
liususan091219
pushed a commit
to liususan091219/sutando-fork
that referenced
this pull request
May 28, 2026
…onichi#263) Fixes a false positive in the voice-transport probe (sonichi#258). Observed live 2026-04-09 at 03:30: voice-agent sat idle for ~10 minutes, then: 03:29:46 [VoiceSession] GoAway from Gemini (timeLeft=50s) 03:30:46 [VoiceSession] Transport closed (state=ACTIVE code=1011 reason="The service is currently unavailable.") 03:30:46 [VoiceSession] Gemini disconnected — will reconnect fresh when client connects That's Google's normal idle-timeout path: Gemini Live fires a GoAway 60s warning, then closes the transport with code=1011 and a "service is currently unavailable" reason. The bodhi session transitions to CLOSED and waits for the next client connect. When a client comes back, `handleClientConnected` reconnects fresh. Not a failure, just the expected lifecycle for an idle voice-agent. The probe was flagging every 1011-after-idle as `fail` with "unrecovered transport close" — which trains users to ignore the signal exactly when we need them to trust it (real 1011s from sonichi#259 duplicate-tool or sonichi#262 googleSearch misconfig get lost in the noise). Fix: track a `goaway_before_close` flag that flips true on GoAway and false on setup complete. Abnormal closes while the flag is true get reclassified as expected idle-timeouts (not flagged). The flag also clears on setup complete so a RECOVERED idle timeout followed by a fresh unrelated 1011 still correctly fails. Test plan: - ✓ Live probe on current log (has idle-timeout 1011 from 03:30): voice-transport ok - ✓ 4-case simulated-log harness: 1. GoAway → 1011 "service unavailable" unrecovered → ok (was fail) 2. Unrecovered 1011 quota, no GoAway preceding → fail (as before) 3. GoAway → 1011 → setup complete (reconnected) → ok 4. Recovered idle-timeout + fresh 1011 with no new GoAway → fail (proves the flag correctly resets on setup complete) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes a visibility gap exposed by the Gemini 3.1 dry-run earlier today. When I unpinned `VOICE_NATIVE_AUDIO_MODEL` to `gemini-3.1-flash-live-preview`, bodhi hit close code 1011 "exceeded your current quota" within 12 seconds of boot. Every existing health check still said ok: port 9900 listening, watchers registered (#250), bodhi uptime normal, startup banner present. The probe layer couldn't see the upstream transport failure. I found it only by manually tailing `voice-agent.log`.
This PR adds `check_voice_transport` that scans the log from the most recent `Sutando — Voice Interface` banner forward and reports the tail state of the Gemini transport.
Reports
Healthy close codes (don't flag)
Failure modes it catches
Wiring
Placed right after `check_voice_watchers` so the voice-agent block in the health report is contiguous:
```
✓ voice-agent ok port 9900
✓ voice-watchers ok all 3 watchers active
✓ voice-transport ok no recent transport errors
```
Same degradation as `check_voice_watchers`: `warn` when voice-agent is down, `warn` if log is missing, `warn` if no banner found.
Test plan
Related
🤖 Generated with Claude Code
Fixes #390