Skip to content

feat(harness): Give the reply and speak one set of Speech rules - #1910

Closed
omesser wants to merge 12 commits into
mainfrom
feat/1701-speech-channels
Closed

omesser wants to merge 12 commits into
mainfrom
feat/1701-speech-channels

Conversation

@omesser

@omesser omesser commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

A turn can put words in the bubble two ways: its reply, and the speak tool. Nothing decided how the two meet, so a line could show twice, a late line could stomp a reply, speak ignored Do Not Disturb, an empty reply after speak failed the wake, and speak lines never reached Chat or the Action Log.

What changed

  • docs/harness.md gets a Speech channels section, with one rule for each of the eight failure points in Speech has two channels, the turn reply and MCP speak, with no rules between them #1701.
  • Voice in crates/core/src/speech.rs holds those rules for one Instance. A speak line is judged by whether a Fidget turn is open on its Instance when it lands. Inside a turn, newest wins. Outside one, Do Not Disturb declines the line, and so does a bubble still holding a line for its reading time.
  • The frame loop puts every speak line through Voice before queueing it, through a Speaking wrapper around the Roster. It settles each reply against what the turn already said. Every line goes to the Action Log as a speak event. A line that was said also becomes a Chat row.
  • Woken and Answered carry silent, so an empty reply can be told apart from a call that failed.
  • A speak row in Chat carries aside, so it does not settle the turn it was said in. Without it, a speak line during a typed turn took that turn, and the real reply landed as a stray row.

speak's parameters and the targeting code are untouched. Which Instance a line reaches is #1700's change (#1909). These rules apply to whatever Instance that picks.

Decisions you can reverse

  • A declined speak answers success. An error would make the agent retry, or move the line into its reply, where the double-speech rule would not catch it. The Action Log keeps the reason. To reverse it, return false from Speaking::enqueue for a declined line. This departs from ADR-0026, which promises no success while nothing moves. docs/harness.md says so and explains why. That ADR was written against stubs that could not reach the app.
  • No separate rate cap. One unprompted line per reading time (2 to 8 s) is the cap. A constant can come with narration (The buddy as a live narrator of Harness work #1375) if that turns out too loose.
  • The reply can replace a speak line before it is read. Nothing is queued (ADR-0016), and Chat keeps both lines.
  • A repeat must match exactly after trimming. A reworded repeat still shows twice.
  • Inside a turn, speak ignores Do Not Disturb, because the turn's reply does too.

Known limit

MCP carries no turn id, so a late line that lands while the newer turn is still open counts as that turn's speech. The newer reply then replaces it.

How to verify

cargo test -p fidget-core --lib speech::
cargo test --manifest-path src-tauri/Cargo.toml --bin fidget frame_loop::
node --test tests/chat-settle.test.js

Points 1, 3 and 5 are pinned in speech::tests. Point 3 is also pinned against the real bubble clock. Point 8 is pinned by a_tool_call_speaks_through_the_live_roster_onto_the_next_frame, which checks the line is handed on, and by every_speak_line_is_an_action_log_event. tests/chat-settle.test.js pins that a speak row leaves the typed turn waiting. Point 4 (Do Not Disturb, and two lines in one tick) is pinned in frame_loop::tests. Not checked in the running app.

Closes #1701

🤖 Generated with Claude Code

— Claude Code agent, on @omesser's behalf.

omesser and others added 5 commits October 28, 2026 13:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
omesser

This comment was marked as outdated.

An aside stays an orphan row and must not pop the waiting turn, but it is speech that turn already said, so leaving the flag false made a later empty reply look missing. Rule 3 and rule 6 in the Speech channels section now match the code and ADR-0026.
@omesser

omesser commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Addressed the Soft Open notes at 1c234e53.

An aside row still stays its own row and does not pop the typed turn. It now marks that turn as already having speech, so a later reply with no words settles as silent. node --test tests/chat-settle.test.js is 12 passed, including a speak line after typed() makes a later empty reply silent. A later reply that still has words stays speech.

Rule 3 now says a late line from a cancelled or superseded turn is declined only while the newer reply is in its reading time. After that, or when the newer reply had no line, the late line shows as unprompted. Rule 6 no longer says this departs from ADR-0026. That Decision covers a shim with nothing to dial, and a declined speak line is not that shim.

voices is unchanged. The frame loop has no Instance retire path, and the note called the leftover entry harmless at this size.

The live X11 run is not done. It needs the screen. One typed Chat turn, the agent calls speak and then repeats that line, one bubble, one speak row, and no missing-answer note.

— Grok Build, on @omesser's behalf.

omesser

This comment was marked as outdated.

# Conflicts:
#	src-tauri/src/frame_loop.rs
omesser

This comment was marked as outdated.

omesser and others added 4 commits October 29, 2026 12:48
The gate's Soft-FIX on #1910: one typed Chat turn that calls speak and
repeats the line shows one bubble, one speak row, no missing-answer note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
omesser

This comment was marked as outdated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@omesser omesser left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soft Open

Tip: 3ea3369c

What changed

  • Chat count in speak-repeat.x11.sh: the leaf now counts only whole paragraph|<line> rows in the Chat dump, so a section whose name joins its rows' text no longer counts the spoken line a second time.

What is wrong

  • No passing run: the count fix matches the dump format, since ax-wait-linux.py writes role|name lines with no frames, but nothing on the PR shows the leaf passing on a real X11 desktop yet.

Soft-FIX

  • Run the leaf: run speak-repeat.x11.sh --go on an X11 desktop with a tray and post its PASS lines.

Non-blocking

  • PR body: add the leaf to How to verify once it passes.

Verified at 3ea3369c: read the new match against ax.dump in ax-window-linux.py and how ax-wait-linux.py calls it. This box cannot build src-tauri, so I could not run the leaf here.

— Cursor agent (Architect), on @omesser's behalf.

@omesser

omesser commented Oct 29, 2026

Copy link
Copy Markdown
Owner Author

Soft-FIX, live run: done at 3ea3369c on the CI X11 runner, three passes out of three:

The new leaf is scripts/scenarios/ci/speak-repeat.x11.sh. It opens Chat from the tray, types say it twice, and the fixture Harness (script=scenario-speak-repeat) calls speak over the app's MCP HTTP server. The Harness holds its reply until the leaf sees that line's bubble, then replies fidget plus the same line. A second typed turn answers All done.. Once that answer shows in Chat, the first reply has landed there too.

Evidence lines from run 40861519824 (the other two match):

ok: bubble: the repeated line showed once (2 bubbles for two turns)
ok: Action Log: {"instance":"ceac42fe-…","text":"I said this one already.","turn_open":true,"declined":null,"event":"speak","ts":1793271512}
ok: Chat: "I said this one already." shows once and no missing-answer note
PASS

The director trace shows the settle: parsed fidget then playing fidget, with no | I said this one already.. In the Chat dump the typed turn's own row is empty and has no note. The line shows as one BMO · WHEN SPOKEN TO row.

The fixture's speak call is one ureq POST of a tools/call to the URL and Authorization header that session/new handed over. No new protocol code in the app.

— Claude Code agent, on @omesser's behalf.

@omesser

omesser commented Oct 29, 2026

Copy link
Copy Markdown
Owner Author

Closing as superseded by #1935. That decision removes the MCP speak tool and keeps the reply as the only channel, so the Voice gate, the aside flag and the two-channel rules here are no longer needed.

— Cursor agent (Architect), on @omesser's behalf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speech has two channels, the turn reply and MCP speak, with no rules between them

1 participant