Repository navigation
fix(voice-agent): close idle Gemini transport to stop phantom turns - #602
Conversation
Voice-agent kept the Gemini Live session alive even with `clientConnected=false`. Every ~9-min Gemini reconnect ("GoAway") then produced a phantom assistant turn 2-3s later — sometimes a tool call (`save_meeting_note`, `open_url`, `work`) that polluted notes files, opened browser tabs, or wrote phantom task files. Reproduced 100% on Studio + verified on Maddy.
Adds a 60s idle teardown:
- on `handleClientDisconnected`, schedule a timer
- timer fires `transport.disconnect()` if no client returned in time
- on `handleClientConnected`, cancel any pending timer
- also armed at boot so a freshly-started voice-agent with no client doesn't keep an idle session open
CLOSED is a fixed point when `clientConnected=false` (existing health monitor only resumes CLOSED→CONNECTING when a client is present), so no phantoms can fire until a real client reconnects.
Verification (post-patch):
- 15:10:50 boot teardown fired, state CLOSED
- 15:11:20–15:13:51 stayed CLOSED through 4 would-be reconnect windows, zero phantoms
- 15:13:54 client connected → Gemini reopens automatically
- 15:14:59 disconnect teardown fired exactly 60s after client gone, state CLOSED
Pre-patch baseline: 90 min idle = 10 phantoms. Post-patch: 6 min idle = 0 phantoms + 2 confirmed teardown events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sonichi
left a comment
There was a problem hiding this comment.
LGTM — addresses today's lifecycle bug cleanly.
Verified architecture:
IDLE_TEARDOWN_MS = 60_000(60s) timer wrapshandleClientDisconnectedto schedule teardown, wrapshandleClientConnectedto cancel it.- Initial arm at boot (line 1167) — voice-agent boots with no client; teardown fires if none connects in 60s. Closes the GoAway-without-client phantom-loop window before it can re-trigger.
- Setting timeout callback re-checks
clientConnectedimmediately before disconnect — prevents race where a client connects between schedule and fire. ✓ cancelIdleTeardown()clears + nulls the handle, idempotent.- The CLOSED-as-fixed-point reasoning is correct: existing health monitor only reconnects CLOSED→CONNECTING when a client is present; without a client, the session stays closed.
Empirical context: Mini's voice-agent had 3 GoAway events clustered in 62s today (logs/voice-agent.log) — exactly the bug-shape this fixes. After this lands + bridge restarts pick up the new code, those clusters should disappear from the logs across the fleet.
Gate check (per repo norm: import strategy / defaults / bind-address):
- Import strategy: no new imports ✓
- Defaults: 60s timer is conservative; first-connect adds at most ~1-2s reconnect latency if user takes >60s post-boot to connect ✓
- Bind-address: N/A (pure timer logic) ✓
One non-blocker note: IDLE_TEARDOWN_MS is a hardcoded const. If we later observe the 60s default is too aggressive (e.g. some user pattern reconnects every ~70s), tuning means a recompile. Could expose as process.env.SUTANDO_VOICE_IDLE_TEARDOWN_MS ?? 60_000. Not in scope; defaults are sensible for the current evidence.
Approve. Recommend merging via gh pr merge --squash (no --delete-branch per repo norm).
… env Per Mini's #602 review non-blocker note: hardcoded 60s default is fine for current evidence, but tuning means a recompile. Read from `process.env.SUTANDO_VOICE_IDLE_TEARDOWN_MS` with the same 60s default — hosts with different connect/disconnect churn can adjust without code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Hi @liususan091219 — Chi wants to merge this. Anything you want to address first, or ready for merge? |
…closes (#617) Issue When the Gemini Live transport closes for a non-retryable reason (1011 credits depleted, 1011 quota exceeded, 401/403 auth, model not found), the voice-agent's health monitor keeps triggering reconnect every 60s. Each reconnect attempt fails after a 30s timeout, so the cycle is ~6 log lines per minute indefinitely while the upstream issue persists. Reproduced live this session (Studio): 17 instances of `code=1011 reason="prepayment credits depleted"` in a 5-min window while clientConnected=true. The voice-failure classifier already fires the OS notification + writes the actionable proactive-result file once per category, so the user has been told. The retry-loop just spams logs after that. PR #602's idle teardown only handles the *no-client* case (60s no client → close). Here a client is connected throughout, so #602 doesn't apply. Fix New `voiceFatalBackoffUntil` deadline (single closure-scoped variable, same scope as `lastReconnectAt`): - Bumped 5min into the future on every non-retryable close, including repeats of an already-notified category. Lives next to the existing notification dedup so the bump fires even when the per-category notification is suppressed. - Read by the 30s health monitor's reconnect-trigger condition — the monitor skips its `handleClientConnected()` call when the deadline is in the future. Reconnect cycle drops from 60s → 5min during a persistent outage. - Reset to 0 when the monitor observes state=ACTIVE — recovery from a transient close after the user fixed the upstream issue is not suppressed by a stale fatal window. No new env vars, no new constants exposed; one local `let` + four arithmetic uses. Defaults conservative (5min ≈ user-visible recovery delay after top-up). Verification - `npx tsc --noEmit` — clean - `npm test` — 154/154 ts + all py green - Trace through the 1011-loop scenario: 1. transport closes with 1011 → classifier → handleClose fires 2. retryable=false → bumps voiceFatalBackoffUntil to now+5min, then notifies (first time only — dedup as before) 3. health monitor 30s tick: state=CLOSED, client=true, 60s throttle elapsed, BUT `Date.now() > voiceFatalBackoffUntil` is false → skip reconnect 4. log dedup at line 1209 keeps the [Health] line silent (status unchanged) — no spam 5. ~5min later, voiceFatalBackoffUntil expires → reconnect fires 6. on success: state=ACTIVE → reset voiceFatalBackoffUntil=0, normal 60s throttle restored 7. on continued failure: classifier re-bumps voiceFatalBackoffUntil Why not unconditional 5min throttle Today's 60s throttle handles transient network blips well. Slowing every reconnect to 5min would degrade UX during normal flaps. Better to slow only the cases the classifier already knows are fatal. Why not stop reconnecting entirely on fatal The upstream issue (top-up, key rotation) often resolves without restarting voice-agent. A 5min auto-recovery window keeps the agent self-healing once credits/auth are back. Stopping permanently would require a manual restart. Cc: stack interaction with PR #602 (idle teardown) — both touch voice-agent.ts but in disjoint regions; merge order doesn't matter. Co-authored-by: Vasiliy Radostev <vasiliyr@mac.mynetworksettings.com>
…onichi#602) * fix(voice-agent): close idle Gemini transport to stop phantom turns Voice-agent kept the Gemini Live session alive even with `clientConnected=false`. Every ~9-min Gemini reconnect ("GoAway") then produced a phantom assistant turn 2-3s later — sometimes a tool call (`save_meeting_note`, `open_url`, `work`) that polluted notes files, opened browser tabs, or wrote phantom task files. Reproduced 100% on Studio + verified on Maddy. Adds a 60s idle teardown: - on `handleClientDisconnected`, schedule a timer - timer fires `transport.disconnect()` if no client returned in time - on `handleClientConnected`, cancel any pending timer - also armed at boot so a freshly-started voice-agent with no client doesn't keep an idle session open CLOSED is a fixed point when `clientConnected=false` (existing health monitor only resumes CLOSED→CONNECTING when a client is present), so no phantoms can fire until a real client reconnects. Verification (post-patch): - 15:10:50 boot teardown fired, state CLOSED - 15:11:20–15:13:51 stayed CLOSED through 4 would-be reconnect windows, zero phantoms - 15:13:54 client connected → Gemini reopens automatically - 15:14:59 disconnect teardown fired exactly 60s after client gone, state CLOSED Pre-patch baseline: 90 min idle = 10 phantoms. Post-patch: 6 min idle = 0 phantoms + 2 confirmed teardown events. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sonichi#602): expose IDLE_TEARDOWN_MS via SUTANDO_VOICE_IDLE_TEARDOWN_MS env Per Mini's sonichi#602 review non-blocker note: hardcoded 60s default is fine for current evidence, but tuning means a recompile. Read from `process.env.SUTANDO_VOICE_IDLE_TEARDOWN_MS` with the same 60s default — hosts with different connect/disconnect churn can adjust without code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Xueqing Liu <xueqingliu@Xueqings-Mac-Studio.local> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…closes (sonichi#617) Issue When the Gemini Live transport closes for a non-retryable reason (1011 credits depleted, 1011 quota exceeded, 401/403 auth, model not found), the voice-agent's health monitor keeps triggering reconnect every 60s. Each reconnect attempt fails after a 30s timeout, so the cycle is ~6 log lines per minute indefinitely while the upstream issue persists. Reproduced live this session (Studio): 17 instances of `code=1011 reason="prepayment credits depleted"` in a 5-min window while clientConnected=true. The voice-failure classifier already fires the OS notification + writes the actionable proactive-result file once per category, so the user has been told. The retry-loop just spams logs after that. PR sonichi#602's idle teardown only handles the *no-client* case (60s no client → close). Here a client is connected throughout, so sonichi#602 doesn't apply. Fix New `voiceFatalBackoffUntil` deadline (single closure-scoped variable, same scope as `lastReconnectAt`): - Bumped 5min into the future on every non-retryable close, including repeats of an already-notified category. Lives next to the existing notification dedup so the bump fires even when the per-category notification is suppressed. - Read by the 30s health monitor's reconnect-trigger condition — the monitor skips its `handleClientConnected()` call when the deadline is in the future. Reconnect cycle drops from 60s → 5min during a persistent outage. - Reset to 0 when the monitor observes state=ACTIVE — recovery from a transient close after the user fixed the upstream issue is not suppressed by a stale fatal window. No new env vars, no new constants exposed; one local `let` + four arithmetic uses. Defaults conservative (5min ≈ user-visible recovery delay after top-up). Verification - `npx tsc --noEmit` — clean - `npm test` — 154/154 ts + all py green - Trace through the 1011-loop scenario: 1. transport closes with 1011 → classifier → handleClose fires 2. retryable=false → bumps voiceFatalBackoffUntil to now+5min, then notifies (first time only — dedup as before) 3. health monitor 30s tick: state=CLOSED, client=true, 60s throttle elapsed, BUT `Date.now() > voiceFatalBackoffUntil` is false → skip reconnect 4. log dedup at line 1209 keeps the [Health] line silent (status unchanged) — no spam 5. ~5min later, voiceFatalBackoffUntil expires → reconnect fires 6. on success: state=ACTIVE → reset voiceFatalBackoffUntil=0, normal 60s throttle restored 7. on continued failure: classifier re-bumps voiceFatalBackoffUntil Why not unconditional 5min throttle Today's 60s throttle handles transient network blips well. Slowing every reconnect to 5min would degrade UX during normal flaps. Better to slow only the cases the classifier already knows are fatal. Why not stop reconnecting entirely on fatal The upstream issue (top-up, key rotation) often resolves without restarting voice-agent. A 5min auto-recovery window keeps the agent self-healing once credits/auth are back. Stopping permanently would require a manual restart. Cc: stack interaction with PR sonichi#602 (idle teardown) — both touch voice-agent.ts but in disjoint regions; merge order doesn't matter. Co-authored-by: Vasiliy Radostev <vasiliyr@mac.mynetworksettings.com>
Summary
Voice-agent kept the Gemini Live session alive even with
clientConnected=false. Every ~9-min Gemini reconnect (GoAway) produced a phantom assistant turn 2-3s later — sometimes a tool call (save_meeting_note,open_url,work) that polluted notes files, opened browser tabs, or wrote phantom task files. Reproduced 100% on Studio + verified on Maddy (latest1c350fa/ bodhiffec0cd).Adds a 60s idle teardown:
handleClientDisconnected, schedule a timertransport.disconnect()if no client returned in timehandleClientConnected, cancel any pending timerCLOSEDis a fixed point whenclientConnected=false(existing health monitor only resumesCLOSED → CONNECTINGwhen a client is present), so no phantoms can fire until a real client reconnects.Pre-patch evidence
Cross-host (Maddy): same pattern, silent-failure surface — phantom
[Tool] save_meeting_notepollutingnotes/meeting-2026-05-04.md(17 lines, all phantom — confirmed by Susan she didn't dictate any).Post-patch verification (Studio, latest main + this patch)
Pre-patch baseline: 90 min idle = 10 phantoms.
Post-patch: 6 min spanning boot + connect + disconnect cycle = 0 phantoms + 2 confirmed teardown events.
Test plan
Client connected → Gemini inactive — resetting session and reconnecting)🤖 Generated with Claude Code