Skip to content

fix(muse): a slow steer acknowledgement no longer fails the run - #24326

Open
only21mil wants to merge 4 commits into
pingdotgg:mainfrom
only21mil:fix/muse-slow-steer-ack
Open

only21mil wants to merge 4 commits into
pingdotgg:mainfrom
only21mil:fix/muse-slow-steer-ack

Conversation

@only21mil

@only21mil only21mil commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When a Muse turn is steered (a user clicks Steer, or a delegated task finishes and T3 steers its result in), Muse acks turn/steer only once the steer's records are on disk. On a busy disk that ack can take more than 30 seconds. The adapter sent the steer through the same 30-second request deadline as every other command, treated the late ack as a dead host, closed Muse and failed the active run. I lost a long Muse run this way: the ack came 33 seconds after the steer.

Muse Code 1.4.12 can also give up on the ack itself. After 30 seconds it answers turn/steer with -32603 internal, "runtime submit failed: runtime command acknowledgement timed out after 30s", and then applies the steer anyway. The adapter failed the steer effect on that error, and the server retried it with a fresh command id, so Muse could run the same steer twice.

Change

turn/steer keeps its 30-second wait but no longer closes the host when the wait runs out. A steer still pending after the wait is reported delivered, and Muse's own "runtime command acknowledgement timed out" error is handled the same way, so the server does not retry it. Every other steer error still fails as before. Every other Muse command keeps the host-closing deadline.

Why "delivered" is the right answer for a pending ack:

  • Muse refuses a steer it will not take (missing_run, invalid_target) within milliseconds, so a refusal never reaches the deadline.
  • Muse keeps an accepted steer even across a Stop and runs it as a new turn, so a late ack still means the message arrives.
  • A retry is what duplicates the steer. Reporting it delivered is the only answer that cannot.

The wait stays bounded at 30 seconds because Stop and approval answers queue behind the steer effect. This change does not touch Stop. On current main, a Stop that Muse does not answer within 15 seconds shows "Agent not responding to Stop" with Force stop.

To do this, the request helper is split into send (no deadline) and request (send plus the host-closing timeout). Only the steer path calls send directly.

Scope and approval

There is no prior issue. This is a small, focused fix for an obvious bug: a slow acknowledgement from a healthy Muse host kills the user's active run. The change is limited to the Muse adapter's steer path and its tests. No other provider, contract or client code changes.

Verification

I ran this signed in to a Muse account in a fresh isolated home: Muse Code 1.4.15, Muse Spark 1.2 at Minimal, Full access. Each build ran as a T3 dev server on Linux ("Test machine", synthetic repo), with every click in T3's Browser panel. Builds: main 0f13d2c, this head bd788bd, and, for the Stop claim only, this head merged locally into that main (a local merge, never pushed), because this head's base predates main's 15 s Stop change. Muse runs sleep 150 and I steer it. Right before Steer, I injected a delay into Muse's own process with strace fault injection: each Muse thread's next fsync/fdatasync waits 40 s, so Muse acks the steer about 80 s late. The slow ack comes from that injected delay, not from Muse on its own.

Claim main This head
A slow steer ack no longer fails the run Steer fails at 30.0 s, run fails Steer reported delivered at 30.0 s, run finishes
The host stays up and Muse gets the steer once Host gets SIGTERM, then SIGKILL, before Muse admits the steer Muse admits the steer once, 80 s after it was sent
Stop while the ack is held Stop waits, the steer fails at 30 s, run fails Merged with main: Stop waits for the steer, shows "Agent not responding to Stop", then ends "Stopped by user" when Muse answers

main: the run fails 30 seconds after the held steer

This head: the run finishes after the late steer; Muse's own background turn follows

On this head Muse later started a turn of its own ("Background activity updated", no user message) that repeated the last line. Muse got the steer once.

main: Stop during the held steer, the run fails

Merged: Agent not responding to Stop, with Force stop (not pressed)

Merged: the run ends Stopped by user

T3 trace, main (second line: the Stop run):

$ grep steerTurn main-trace.txt | tr -s ' '
04:28:52.363Z -> 04:29:22.367Z 30003.5 ms Failure MuseAdapter.steerTurn [Muse turn/steer timed out; its host was closed.]
04:30:23.877Z -> 04:30:53.880Z 30003.3 ms Failure MuseAdapter.steerTurn [Muse turn/steer timed out; its host was closed.]

This head (lines 2 and 3: its own Stop run; its base predates main's Stop change):

$ grep -E 'steerTurn|interruptTurn' head-trace.txt | tr -s ' '
04:34:16.874Z -> 04:34:46.875Z 30000.3 ms Success MuseAdapter.steerTurn
04:38:10.290Z -> 04:38:40.292Z 30001.6 ms Success MuseAdapter.steerTurn
04:38:40.309Z -> 04:39:10.317Z 30008.2 ms Failure MuseAdapter.interruptTurn [Muse turn/interrupt timed out; its host was closed.]
Muse's events and the shutdown on main

Muse's events, main (no steer):

$ sed -n '2p;4,5p' main-muse-events.txt | tr -s ' '
04:28:08.449 session/started 
04:28:08.614 turn/started 
04:28:08.614 item/completed userMessage <T3 runtime preamble + user prompt>

This head:

$ sed -n '4,7p;11,12p' head-muse-events.txt | tr -s ' '
04:34:04.574 turn/started 
04:34:04.574 item/completed userMessage <T3 runtime preamble + user prompt>
04:35:36.944 item/completed userMessage steered=true HEAD-STEER-H1
04:37:37.226 item/completed agentMessage FINISHED HEAD-STEER-H1
04:39:27.211 turn/started 
04:39:30.220 item/completed agentMessage FINISHED HEAD-STEER-H1

main, strace (the test host's local clock, six hours behind the UTC lines above):

$ sed -n '1p;3p;6,8p' main-strace-M1.txt
2584141 22:28:52.366088 fdatasync(12 <unfinished ...>
2584141 22:29:32.368997 <... fdatasync resumed>) = 0 (DELAYED)
2584141 22:29:32.375379 fsync(12 <unfinished ...>
2584131 22:29:32.375718 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=2583051, si_uid=1000} ---
2584145 22:29:34.376200 +++ killed by SIGKILL +++

Tests, main with this PR's test file, then this head:

$ grep -E '^ +(×|Tests )' main-tests.txt
     × reports a steer delivered and keeps the host while Muse holds its ack 24ms
     × reports a steer delivered when Muse times out its own ack 5ms
      Tests  2 failed | 49 skipped (51)
$ grep -E '^ +Tests ' head-tests.txt head-provider-muse-tests.txt
head-tests.txt:      Tests  2 passed | 39 skipped (41)
head-provider-muse-tests.txt:      Tests  91 passed (91)

Full logs and test output: transcript

Not checked: Muse's own "acknowledgement timed out" answer (T3's wait always fired first; the second test covers it), macOS and Windows, desktop and mobile, and a server restart between a steer and its admission.

Prior PRs

Generated with Claude Opus 5.5 in T3 Code Nightly

only21mil and others added 2 commits October 23, 2026 17:18
Muse acks an accepted turn/steer only once the steer's records are on
disk. On a slow disk that ack can take more than 30 s, and Muse Code
1.4.12 may also answer with its own -32603 internal "runtime command
acknowledgement timed out after 30s" error and then apply the steer
anyway.

The adapter treats both as failures: its 30 s request timeout closes the
host and fails the active run, and Muse's error fails the steer effect,
which the server retries with a fresh command id, so Muse runs the steer
twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Muse refuses a steer it will not take within milliseconds, and keeps an
accepted steer even across a Stop, running it as a new turn. So turn/steer
now waits up to the request timeout and, if the ack is still pending,
reports the steer delivered and keeps the host. Muse's own "runtime
command acknowledgement timed out" error is handled the same way, so the
server does not retry it. Every other steer error still fails.

The wait stays bounded at the request timeout because Stop and approval
answers queue behind the steer effect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The Muse adapter separates raw RPC sending from the timeout-handling request wrapper. Turn steering uses the raw send path and handles missing acknowledgements and a specific runtime-command acknowledgement timeout differently from other errors. Tests cover steer and interrupt timeout behavior.

Changes: Muse steer acknowledgement handling

Layer / File(s) Summary
Separate RPC sending from timeout handling
packages/provider-muse/src/server/adapter.ts
The adapter adds a raw send helper. The request wrapper retains the configured timeout and timeout-triggered host failure behavior.
Handle steer acknowledgements
packages/provider-muse/src/server/adapter.ts, packages/provider-muse/src/server/adapter.test.ts
steerTurn uses send and returns when no acknowledgement arrives or Muse reports the specific runtime-command acknowledgement timeout error. Other errors propagate, and received acknowledgements are checked against the active turn. Tests cover steer and interrupt timeout outcomes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related PRs

  • pingdotgg/t3code#21884: Changes per-method deadlines in the Muse request wrapper, which this PR splits into separate send and request paths.

Suggested reviewers: t3dotgg

Merge Risk: Minimal · up to bd788

The change appears mergeable after normal checks. Slow steer acknowledgements no longer fail the run, while other requests retain their timeout handling.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main fix: slow Muse steer acknowledgements no longer fail the run.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is complete and follows the required template. It explains the problem, expected behavior, implementation, scope, approval rationale, focused verification, observed results, limitation…
Full details: Description check

Explanation

The description is complete and follows the required template. It explains the problem, expected behavior, implementation, scope, approval rationale, focused verification, observed results, limitations, conflicts, and the model and harness used.

Finishing Touches

🧪 Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 572cf74 and bd788bd.

Files selected for processing: 2

  • packages/provider-muse/src/server/adapter.test.ts
  • packages/provider-muse/src/server/adapter.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 24, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 2a545ab

Macroscope's review found this PR approvable — This narrowly scoped Muse adapter fix preserves the existing timeout behavior for all other commands while preventing delayed steer acknowledgements from closing the host or failing the run. Targeted tests cover both delayed responses and Muse’s specific acknowledgement-timeout error.

You can add or adjust custom eligibility rules. Learn more.

@maria-rcks

maria-rcks commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

@only21mil, thanks for isolating the durable-write acknowledgement timeout and keeping other errors strict. i kept both original commits and merged current main (572cf74275) into this branch. the fix retains your send/request split and narrow internal-error exception. i shortened the explanatory comments, restored the Option import removed on main, and strengthened the existing slow-ack test to cover a late response, one steer submission, and a subsequent wedged Stop closing the host.

verification on bd788bd4827ab84dd8ea6ded434bc268aa778d1b:

  • both acknowledgement regressions fail with current main's adapter, including the host-closing timeout.
  • all 91 muse tests pass through blacksmith. vp run check:changed passes format, lint, knip, five package typechecks, and all 41 adapter tests. scoped muse typecheck and lint also pass.
  • two independent full-diff reviews found no blocking code issues. known test limit: the late-response call-count assertion precedes promise continuation processing; the SDK's maxAttempts: 1 remains the primary no-retry guarantee.
  • current-main red/green and check output.

unverified: the real muse provider/client path, delayed production fsync, and the production error payload. this vm has no muse cli or production blob, so the controlled adapter tests establish t3's behavior, not the provider's eventual delivery guarantee. no client or ui implementation changed; no new app capture is claimed. @only21mil, please retain a follow-up for the documented queued-resume interaction with #24325; it does not expand this focused takeover.

ci build/typecheck, lint, web tests, all eight server shards, and the aggregate Check are green on this head. automatic CodeRabbit review completed on this head with no actionable comments. github reports the pr clean and mergeable. macroscope approval is from the original head; no fresh macroscope run was emitted.

maria-rcks commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

Verified in a real dev environment

i ran current github/main (9b052a2d5b) in a separate worktree, stopped it, then ran exact pr head bd788bd4827ab84dd8ea6ded434bc268aa778d1b. both installs used vp i. each dev server ran in the background with isolated state, with only one running at a time. i drove the paired web app with repo playwright and /usr/bin/google-chrome.

i created a project through the app pointing at a real git repository. on the pr head, real codex gpt-6.1-sol turns verified these nearby behaviors:

  • send: the provider ran sleep 75 through its terminal tool.
  • steer: while that command ran, i clicked steer. the completed turn replied STEER_ACK_24326 once, replacing the original requested reply.
  • stop: i started another sleep 120, clicked stop while it ran, and saw "run interrupted · stopped by user".
  • follow-up: the same thread then replied RESUME_OK_24326 without resuming either command.

these are codex regression checks. they do not execute the changed muse adapter.

pr head: real codex steer, stop, and follow-up replies

i inspected the captures and reopened this comment to check the screenshots. the 14-second stop recording is shared in the task thread; playback and seeking work in t3's browser, but github does not render the external mp4 inline. runtime command transcript.

Not verified

the changed muse behavior is unverifiable on this vm, not verified by the codex checks above. the vm initially had no muse executable. i installed the real public linux binary, muse 1.4.14-R7309.1, and checked its sha256 against the vendor manifest. it runs, but muse usage reports "not logged in", and a real muse exec attempt fails with "missing meta credentials". no meta credential is configured. adding muse through settings in both running apps reports zero models and asks for muse login.

current main: muse login required and zero models

exact pr head: the same muse authentication blocker

i therefore could not reproduce the original timeout on main or prove the pr's delayed acknowledgement, muse's own acknowledgement-timeout error, eventual delivery without duplication, or wedged-muse stop behavior. those need an authenticated real muse account and delayed durable-write acknowledgement. no mocks, fabricated catalog, echo provider, or fixtures were used to replace that path.

native desktop, ios/android, macos/windows, delegated-task completion, and the documented queued-resume interaction were not exercised.

the existing github aggregate Check passes on this head. no new automated-check result is claimed. i made no source changes, commits, or pushes. this pass does not establish that the muse fix works end to end.

@only21mil

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of only21mil

@maria-rcks, retained. The queued-resume interaction with #24325 stays a separate follow-up and is not added to either PR.

What the interaction is, from reading both heads (bd788bd482 here, 64269818ee on #24325):

  1. Muse holds a steer's ack past 30 seconds. This PR reports the steer delivered, so T3 does not send it again.
  2. The server restarts before Muse admits the steer, so the steer is still queued in Muse.
  3. On resume with session MCP configured, Muse replays the queued steer and the resume ack names an active turn. fix(muse): a resume that replays queued work keeps T3 tools #24325 then closes that host and resumes on a fresh one, which cancels the replayed steer.

The steer is lost while T3 has recorded it as delivered. This needs both PRs merged and a restart between the late ack and Muse admitting the steer. Each PR is correct on its own.

Not verified: no test or real Muse run has exercised this sequence. It comes from the code and matches the overlap both PR bodies describe. Your runtime note also lists it as not exercised.

My judgment, not verified: the fix belongs in #24325's relaunch path rather than here, because reporting a pending ack as delivered is what keeps a steer from running twice.

@only21mil

Copy link
Copy Markdown
Contributor Author

Note

🤖 Opus 5.5 and Sol 6.1 replying on behalf of Victor.

Re your comment: the queued-resume overlap doesn't need a follow-up on this PR. #24325's relaunch is what cancels the replayed steer. This PR decides whether a slow steer ack lasts until a restart, and doesn't change what #24325 does on resume.

I merged #24325's head (9e25000a9e) into this PR's head (bd788bd482) locally, pushed nothing, and added one adapter test. In the first server lifetime it steers a running turn with session MCP configured, and the ack comes back either at once or after 31 seconds of held time. In the second lifetime, session/resume comes back with activeTurnId set, the way Muse replays a queued steer. The test checks that the replaying host is closed, a fresh host resumes, and nothing sends turn/steer again.

adapter.ts Prompt ack Held ack
main 67eebaca23 + #24325 only replaying host closed on resume steer fails at 30s and the host closes, as on main
main 67eebaca23 + this PR only no relaunch no relaunch
both replaying host closed on resume replaying host closed on resume

With #24325 alone, a steer that was acked promptly but not yet admitted at the restart is cancelled the same way. A held ack never reaches the restart there, because the 30-second timeout fails the steer and closes the host first. With both PRs, both ack timings reach the resume and #24325 closes the replaying host. This PR adds no state to that path. If the one-replayed-item cost needs fixing, the change belongs in #24325's relaunch. I'll correct the "Not checked" bullet in this PR's description, which says the overlap needs both PRs.

Not checked: a real Muse host, or a late turn/steer ack that arrives after the old host has closed. The fake can't show what Muse keeps of a cancelled steer. In #24325's live run, a cancelled queued user message stayed in history.

@only21mil

Copy link
Copy Markdown
Contributor Author

Note

🤖 Opus 5.5 replying on behalf of Victor.

@maria-rcks thanks for the run; we did it with a real signed-in Muse (Muse Code 1.4.15, Spark 1.2 at Minimal, isolated login), steering a sleep 150 turn while strace held Muse's next disk sync for 40 s so the steer ack arrived about 80 s late. On main (d313b2aff3) the steer times out at 30.0 s, the host is closed and the run fails; at bd788bd482, and on this head merged with main locally, the steer is reported delivered at 30.0 s, the host stays up, Muse admits it 80 s after sending with one turn/steer call, and the reply quotes it once. A frozen host never hung Stop on any build (on current main Stop gives up after 15 s and offers Force stop, so we corrected that sentence in the description), and Muse's own "acknowledgement timed out" answer stays covered only by the adapter test because T3's 30 s wait always expired first (run summary).

Main: the delayed steer fails the run at 30 s

This PR: the same steer is admitted late and quoted once

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants