Shutdown-held queues never drain after CLI sends on idle threads #25780

Activity

1. omid3098 commented on Oct 27, 2026

   

   omid3098

   on Oct 27, 2026

   More actions

   Claude Opus 5.5 wrote this for the reporter of the case below.

   Another case of held queued runs that never drain, on 0.1.3-nightly.20261026.3717 (Windows). It involves no CLI send: all of the held runs are automatic delegate_task completion deliveries to a Claude parent thread.

   What the parent's rows in orchestration_v2_projection_runs showed:

   • Run 24, a delegated-task failure notice queued at 2026-10-27T08:53:52Z, stayed queued with queueHeld: true.
   • Four later completion notices came in while the parent was mid-turn, and each one inherited the hold: run 43 (10:49:49Z), run 44 (10:49:54Z), run 46 (11:01:18Z) and run 51 (11:12:57Z). Each was a Delegated task … [completed] delivery for a child that finished during one of the parent's turns.
   • Notices that arrived while the parent was idle started at once (runs 45, 47–50, 52–70). So the thread looked healthy, but it never received those four results. For about four hours its own reports listed the four finished children as still running.
   • t3_queue_list on the parent showed the five held runs. To recover, I cancelled them and passed the results to the parent with t3_thread_send.

   I could not find what set the first hold, because the server trace logs from that morning had already rotated. Around 08:45–08:53Z several delegated children, and a turn in another thread, failed with "Provider turn failed". So the hold may have come from the recovery after that failure, not from a shutdown.

   In this case the cost of the hold was silent: nothing on the parent or in the agent's tools said that its queue was held. A completion delivery that cannot start, or that inherits a hold, should either start once the parent is idle or be visible to the parent and the user as held.
2. EtanHey commented on Oct 28, 2026

   

   EtanHey

   on Oct 28, 2026

   Contributor

   More actions

   Additional read-only evidence collected through the built-in T3 triage playbook, Codex / GPT-6.1 Sol, 2026-10-28. Current desktop and live server: official 0.1.4-nightly.20261028.3814, macOS arm64. The historical initiating build was not independently recorded, so this is not a claim that the initial hold began on .3814.

   An affected thread had 17 never-started held runs, requested between 00:18:16.737Z and 06:21:49.268Z. Its event history gives a concrete first cause: startup reconciliation at 00:45:51.723Z set queueHeld: true on runs 19, 21 and 22 (command:runtime-reconcile:startup:<redacted-thread>:2026-10-28T00:45:51.723Z). Later queued runs inherited the hold. Twelve of the 17 inputs were server-origin notices; five were MCP-origin messages. Newer runs executed while the old ones never started. The old runs were manually cancelled at 09:11:09.764Z–09:11:29.676Z; the oldest waited 31,973.027 seconds before cancellation, without starting.

   The same installed source releases held runs in dispatchMessage only for createdBy === "user" and creationSource === "web" || "mobile" (excluding scheduled tasks). Agent/MCP sends and automatic notices do not satisfy that condition. This explains why a healthy-looking thread and newer successful turns do not establish that its earlier queue drained. The latest published .3824 retains this release predicate; that newer app was not run here.

   A fresh t3_queue(action:"list") at 13:12:44Z also found six old PR-watch notices in another failed thread; its read-only run projections record queueHeld: true. The recovered first thread's queue is now empty.

   Separately, our agents sometimes explicitly sent mode:"queue" while intending to steer. That is our misuse and is not evidence of a T3 steer failure. We did not promote/cancel other work or write the database during this investigation. This case strengthens the recovery/held-queue report rather than alleging that every queued follow-up is broken.
3. t3dotgg commented on Oct 31, 2026

   

   t3dotgg

   on Oct 31, 2026

   Member

   More actions

   Note

   🤖 Claude Opus 5.5 responding on behalf of Theo

   Closing: this is covered by #27221, which landed. Reopen if an important part is still missing.
4. t3dotgg

   closed this as completedon Oct 31, 2026