Skip to content

fix(chat): heartbeat keeps long reasoning turns alive past 2-min timeout (#4270) - #4282

Merged
M3gA-Mind merged 7 commits into
tinyhumansai:mainfrom
oxoxDev:fix/4270-inference-heartbeat-timeout
Jun 30, 2026
Merged

M3gA-Mind merged 7 commits into
tinyhumansai:mainfrom
oxoxDev:fix/4270-inference-heartbeat-timeout

Conversation

@oxoxDev

@oxoxDev oxoxDev commented Jun 29, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Long-running Reasoning-mode turns no longer hit a false "No response from the agent after 2 minutes" timeout that discarded the in-flight turn.
  • The core progress bridge now emits a periodic inference_heartbeat (every 20s) for the whole duration of an in-flight turn.
  • The frontend silence timer rearms on that beat, so a turn that streams no other progress for minutes stays alive — while a genuine disconnect (beats stop) still fails fast.
  • No new user-facing UI; reuses the existing chat.safetyTimeout error for true connectivity failures.

Problem

The 2-minute timeout is a purely frontend silence timer in app/src/pages/Conversations.tsx (armSilenceTimer, 120s). It only rearms when one of four redux slices changes (inferenceStatusByThread, streamingAssistantByThread, toolTimelineByThread, taskBoardByThread). inference_start arms it once at turn start; after that, two legitimate cases produce zero progress events for >120s while the turn is genuinely working:

  1. A long prefill / time-to-first-token on a large context (the report was at 223K/1M tokens).
  2. A reasoning-tier model whose provider buffers reasoning_content server-side and streams no intermediate delta.

In both, the silence timer fires, surfaces chat.safetyTimeout, clears the runtime, and the user loses the turn with no output. No core-side liveness signal existed (the heartbeat module is the unrelated subconscious loop).

Solution

  • Core (channels/providers/web/progress_bridge.rs): convert the bridge's while let event loop to a loop { tokio::select! { … } } with a biased interval branch. Every INFERENCE_HEARTBEAT_SECS (20s) it publishes a WebChannelEvent { event: "inference_heartbeat", … }, gated on turn_active (set on TurnStarted, cleared on TurnCompleted). The beat rides the same socket as real progress, so it stops the instant the socket/core dies.
  • Frontend: subscribe to inference_heartbeat (chatService.ts), bump a per-thread inferenceHeartbeatByThread counter (chatRuntimeSlice.ts) from ChatRuntimeProvider.tsx, and add that counter to the silence-timer rearm signature in Conversations.tsx (kept last so the done-transition probe on current[0] is unaffected).
  • Design note: a fixed/per-mode timeout bump was rejected — it just moves the wall and slows the genuine-disconnect error. A heartbeat is the right layer: 120s now measures heartbeat gaps (6 missed beats), preserving fail-fast on real disconnects.

Submission Checklist

If a section does not apply to this change, mark the item as N/A with a one-line reason. Do not delete items.

  • Tests added or updated (happy path + at least one failure / edge case) per Testing Strategy
  • Diff coverage ≥ 80% — Rust paused-clock tests cover the heartbeat emit + gate; Vitest covers service wiring, slice reducer, rearm liveness, and the beats-stop regression path.
  • N/A — Coverage matrix: internal liveness signal, no new user-facing feature ID
  • All affected feature IDs from the matrix are listed in the PR description under ## Related
  • No new external network dependencies introduced (tokio test-util is a test-only dev-dependency)
  • N/A — Manual smoke: timing-only fix; automated paused-clock + fake-timer tests exercise both repro-gone and regression-safety paths
  • Linked issue closed via Closes #NNN in the ## Related section

Impact

  • Desktop only behavior: in-flight chat turns gain a 20s liveness beat over the existing Socket.IO channel — negligible overhead (one tiny event with no payload).
  • No migration, no schema, no security surface. Backwards-compatible: older FE clients ignore the unknown event; the beat is purely additive.
  • Genuine connectivity failures still surface chat.safetyTimeout within ~120s of the last beat.

Related


AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issue

  • Key: N/A
  • URL: N/A

Commit & Branch

  • Branch: fix/4270-inference-heartbeat-timeout
  • Commit SHA: 5e45cdb6f

Validation Run

  • pnpm --filter openhuman-app format:check
  • pnpm typecheck
  • Focused tests: vitest run Conversations.render / chatService / chatRuntimeSlice (110 passed); cargo test --lib …web::progress_bridge (6 passed)
  • Rust fmt/check (if changed): cargo fmt --check, cargo check, cargo clippy --lib (clean for changed file)
  • N/A — Tauri fmt/check: no src-tauri code changed

Validation Blocked

  • command: N/A
  • error: N/A
  • impact: N/A

Behavior Changes

  • Intended behavior change: long silent reasoning turns are kept alive by a liveness beat instead of false-timing-out at 2 minutes.
  • User-visible effect: Reasoning-mode tasks that take >2 minutes now complete and render instead of showing the timeout error.

Parity Contract

  • Legacy behavior preserved: genuine disconnects still trip the 120s silence timer and surface chat.safetyTimeout.
  • Guard/fallback/dispatch parity checks: rearm signature keeps status at index 0 so the done-transition (chat_done/chat_error) detection is unchanged; heartbeat counter cleared on clearRuntimeForThread.

Duplicate / Superseded PR Handling

Summary by CodeRabbit

  • New Features

    • Added periodic “inference heartbeat” updates during long-running turns so conversation threads stay active even when no other progress is reported.
  • Bug Fixes

    • Prevented the 120-second inactivity/silence timer from prematurely clearing threads during silent reasoning phases.
    • Heartbeats now apply to the primary turn only, not parallel/lane requests.
  • Tests

    • Added end-to-end and unit test coverage to verify heartbeat behavior, timer rearming/clearing, and correct thread targeting.
    • Updated test tooling to use Tokio’s paused time for reliable heartbeat timing.

oxoxDev added 3 commits June 29, 2026 14:51
… turn (tinyhumansai#4270)

The frontend silence timer fires after ~120s with no progress signal of any
kind. A long prefill on a large context, or a reasoning-tier model that buffers
reasoning_content server-side, can legitimately stream nothing for minutes,
tripping a false "no response after 2 minutes" timeout that discards the live
turn.

Add a 20s wall-clock liveness beat ("inference_heartbeat") emitted by the
progress bridge for the whole duration of an in-flight turn (gated on
TurnStarted, stopped on TurnCompleted). It rides the same socket as real
progress events, so it keeps the turn alive while work progresses yet stops the
instant the socket/core dies — preserving the genuine-disconnect error path.

Tokio test-util dev-dependency added so the paused-clock heartbeat tests assert
the beat without real-time waits.
…4270)

Subscribe to the new inference_heartbeat socket event, bump a per-thread
liveness counter in chatRuntime, and add that counter to the Conversations
silence-timer rearm signature. A long buffered-reasoning phase that emits no
status/stream/tool/board signal now keeps the 120s timer armed via the beat,
while a real disconnect (beats stop) still fails fast.
…inyhumansai#4270)

- chatService: inference_heartbeat forwards through onInferenceHeartbeat
- chatRuntimeSlice: heartbeat counter bumps per thread, cleared on runtime reset
- Conversations: beats during a silent reasoning phase keep the turn active past
  the 120s window (repro gone); when beats stop the silence timer still fires
  (genuine-disconnect regression safety)
@oxoxDev
oxoxDev requested a review from a team June 29, 2026 09:27
@coderabbitai

coderabbitai Bot commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 49750fe9-a0f2-44bf-9082-a615ffea2a04

📥 Commits

Reviewing files that changed from the base of the PR and between bf42574 and dac701a.

📒 Files selected for processing (3)
  • Cargo.toml
  • app/src/pages/Conversations.tsx
  • app/src/pages/__tests__/Conversations.render.test.tsx
🚧 Files skipped from review as they are similar to previous changes (3)
  • Cargo.toml
  • app/src/pages/Conversations.tsx
  • app/src/pages/tests/Conversations.render.test.tsx

📝 Walkthrough

Walkthrough

Adds periodic inference_heartbeat events from the Rust progress bridge and threads them through the frontend service, store, provider, and Conversations silence timer.

Changes

Inference Heartbeat End-to-End

Layer / File(s) Summary
Progress bridge heartbeat emission
src/openhuman/channels/providers/web/progress_bridge.rs, Cargo.toml
Defines the 20s heartbeat interval, rewrites the bridge loop to emit inference_heartbeat only while a turn is active, toggles that state on turn start/completion, and adds paused-time tests plus tokio test utilities.
Chat service heartbeat subscription
app/src/services/chatService.ts, app/src/services/__tests__/chatService.test.ts
Adds the heartbeat event type, listener hook, socket mapping, and forwarding logic, with a test that verifies the event is subscribed and passed through unchanged.
Runtime heartbeat counter state
app/src/store/chatRuntimeSlice.ts, app/src/store/__tests__/chatRuntimeSlice.test.ts
Adds the per-thread heartbeat counter, reducer, cleanup paths, and store coverage for incrementing and reset behavior.
Provider dispatch and silence-timer rearm
app/src/providers/ChatRuntimeProvider.tsx, app/src/pages/Conversations.tsx, app/src/providers/__tests__/ChatRuntimeProvider.test.tsx, app/src/pages/__tests__/Conversations.render.test.tsx
Dispatches heartbeat updates from the socket, ignores parallel lanes, folds the counter into Conversations timer rearming, and adds regression tests for active-heartbeat and heartbeat-stop cases.

Sequence Diagram(s)

sequenceDiagram
  participant progress_bridge
  participant socket
  participant provider as ChatRuntimeProvider
  participant store as Redux Store
  participant conv as Conversations.tsx

  progress_bridge->>socket: inference_heartbeat
  socket->>provider: onInferenceHeartbeat(event)
  provider->>store: dispatch bumpInferenceHeartbeatForThread(thread_id)
  store-->>conv: inferenceHeartbeatByThread updated
  conv->>conv: rearm 120s silence timer
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐇 Thump, thump — a heartbeat in flight,
Long quiet turns stay bright tonight.
One tiny pulse, then one more too,
Keeps the rabbit trail alive and true.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Linked Issues check ❓ Inconclusive The heartbeat-based timeout fix matches #4270 and regression tests cover the long-running and failure paths, but coverage gate compliance isn't verifiable here. Provide the changed-lines coverage report or CI evidence showing the PR meets the ≥80% diff coverage requirement.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main fix: adding heartbeat support to keep long reasoning turns alive past the 2-minute timeout.
Out of Scope Changes check ✅ Passed The changes stay focused on timeout prevention, socket handling, state tracking, and tests for the linked issue.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

Comment @coderabbitai help to get the list of available commands.

…nsai#4270)

Lifecycle test: turn beats while in flight, then stops once TurnCompleted lands
and the channel closes — exercises the turn_active=false arm and the loop break
so diff coverage holds on the changed bridge lines.
@coderabbitai coderabbitai Bot added bug rust-core Core Rust runtime in src/: CLI, core_server, shared infrastructure. labels Jun 29, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/src/providers/ChatRuntimeProvider.tsx`:
- Around line 478-484: The heartbeat handling in ChatRuntimeProvider is
collapsing liveness to thread scope, which lets parallel requests on the same
thread incorrectly keep the primary turn alive. Update the onInferenceHeartbeat
path to preserve request scoping using ChatInferenceHeartbeatEvent.request_id,
and make bumpInferenceHeartbeatForThread or the surrounding runtime state filter
by the owning request before dispatching the keepalive. Ensure the primary
silence timer attached by handleSendParallel only gets bumped by its own
request’s heartbeat, not sibling branches.

In `@app/src/store/chatRuntimeSlice.ts`:
- Around line 414-422: Add `inferenceHeartbeatByThread` to the state reset
performed by `clearAllChatRuntime()` in `chatRuntimeSlice`, so the slice is
fully cleared and no stale per-thread heartbeat counters survive. Use the
existing `clearAllChatRuntime` reducer and the new `inferenceHeartbeatByThread`
bucket to locate the change, and update the slice test to assert this map is
reset along with the other chat runtime fields.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5e358638-415b-4062-bb6e-1a8d32560b02

📥 Commits

Reviewing files that changed from the base of the PR and between d867c1f and 2ccaf10.

📒 Files selected for processing (9)
  • Cargo.toml
  • app/src/pages/Conversations.tsx
  • app/src/pages/__tests__/Conversations.render.test.tsx
  • app/src/providers/ChatRuntimeProvider.tsx
  • app/src/services/__tests__/chatService.test.ts
  • app/src/services/chatService.ts
  • app/src/store/__tests__/chatRuntimeSlice.test.ts
  • app/src/store/chatRuntimeSlice.ts
  • src/openhuman/channels/providers/web/progress_bridge.rs

Comment thread app/src/store/chatRuntimeSlice.ts Resolved
oxoxDev added 2 commits June 29, 2026 16:30
…inyhumansai#4270)

CodeRabbit: a parallel (forked) turn shares the thread but streams into its own
lane; its heartbeat must not keep the thread's primary silence timer alive, or a
sibling branch would mask a stalled primary turn. Mirror the text/thinking-delta
routing — skip the bump when the heartbeat's request_id is a registered parallel
request.
…inyhumansai#4270)

CodeRabbit: the slice-wide reset left the new heartbeat bucket behind, so stale
per-thread counters survived a full chat-runtime clear. Wipe it alongside the
other runtime maps + assert in the slice test.
@coderabbitai coderabbitai Bot added the agent Built-in agents, prompts, orchestration, and agent runtime in src/openhuman/agent/. label Jun 29, 2026
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 29, 2026
Conversations.render.test.tsx: both sides added named imports from
chatRuntimeSlice — this PR's bumpInferenceHeartbeatForThread and main's
clearFollowupsForThread/enqueueFollowup. Kept all three (alphabetical); each is
used by its own test block, both of which are present in the merged file.
@coderabbitai coderabbitai Bot removed rust-core Core Rust runtime in src/: CLI, core_server, shared infrastructure. bug agent Built-in agents, prompts, orchestration, and agent runtime in src/openhuman/agent/. labels Jun 30, 2026

@M3gA-Mind M3gA-Mind left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and verified: correct, well-tested, CI green. Approving. Thanks for the contribution!

@M3gA-Mind
M3gA-Mind merged commit ea262ea into tinyhumansai:main Jun 30, 2026
17 of 27 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in Team Openhuman Jun 30, 2026
senamakel pushed a commit to nocstah/openhuman that referenced this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Agent times out after 2 minutes on long-running reasoning tasks

2 participants