Cancelling one queued completion message disables future results from still-running sibling tasks #32806

Activity

1. Dropje97

   changed the title [-][Bug]: Cancelling one queued completion message disables future results from still-running sibling tasks[/-] [+]Cancelling one queued completion message disables future results from still-running sibling tasks[/+] on Nov 8, 2026
2. aron-intframe commented on Nov 8, 2026

   

   aron-intframe

   on Nov 8, 2026

   More actions

   Reproduced on main at a1af1841 with a server test. Passing interruptedWakeTaskIds from the queued-cancel caller is not enough on its own, and the same cancel also drops tasks the coordinator starts after it.

   Cause. dispatchQueuedRunCancel (Orchestrator.ts:10076) calls disposeDelegatedCompletionCohort without interruptedWakeTaskIds, so keptTaskIds is empty (:2814). Every outstanding app-owned task on the owning run is marked disposed, and the cohort itself becomes disposed (:2831). planDelegatedCompletionDelivery then returns offer: false for B because of its row, and for any later task because the cohort is no longer open (:12548).

   How I checked. I copied the "interrupted completion wakes" case (DelegatedCompletionDelivery.test.ts:1383) and changed three things: the coordinator run stays running so A's wake is queued, not started; run.interrupt becomes queued-run.cancel; B is a delegated_task.request sibling that completes after the cancel. A second case starts the task only after the cancel, with no sibling outstanding.

   vp test run apps/server/src/orchestration-v2/DelegatedCompletionDelivery.test.ts -t "queued completion wake" --silent=false
   

   On main:

   [repro] after queued-run.cancel: {"wakeStatus":"cancelled","incoming":"disposed","siblingStatus":"running","siblingDelivery":"disposed","cohort":"disposed"}
   AssertionError: a still-running sibling that was not in the cancelled message must keep its wake: expected 'disposed' to not equal 'disposed'
   [repro] after queued-run.cancel: {"wakeStatus":"cancelled","incoming":"disposed","cohort":"disposed"}
   [repro] late task after it completes: {"lateTaskDelivery":"disposed","cohort":"disposed"}
   Tests  2 failed | 39 skipped (41)
   

   Why the one-line fix falls short. With only interruptedWakeTaskIds: queuedMessage.delegatedCompletion.taskIds, B is kept, but emitQueuedRunCancellation is gated on keptTaskIds.size === 0 (:2876), so the queued message is never removed:

   [repro] after queued-run.cancel: {"wakeStatus":"queued","incoming":"disposed","siblingStatus":"running","siblingDelivery":null,"cohort":"open"}
   AssertionError: expected 'queued' to equal 'cancelled'
   

   Adding cancelQueuedDelivery: false and an explicit emitQueuedRunCancellation fixes the sibling case, but the late-task case still ends lateTaskDelivery: "disposed", because with nothing kept the cohort is still disposed.

   Fix that passes both cases. Removing a queued wake disposes only the tasks that message carried, cancels the run, and leaves the cohort open. The terminal-run listener already handles a cancelled delivery run (:13156): it clears cohort.delivery, or reserves a successor for pending results, so the next result gets a new generation.

   diff --git a/apps/server/src/orchestration-v2/Orchestrator.ts b/apps/server/src/orchestration-v2/Orchestrator.ts
   index 5d3c4391..d832e930 100644
   --- a/apps/server/src/orchestration-v2/Orchestrator.ts
   +++ b/apps/server/src/orchestration-v2/Orchestrator.ts
   @@ -10072,15 +10072,28 @@ const makeOrchestrator = Effect.fn("orchestrationV2.Orchestrator.layer")(functio
          (candidate) => candidate.id === queuedRun.userMessageId,
        );
        if (queuedMessage?.delegatedCompletion !== undefined) {
   +      // Removing a wake drops only the results it carried; the cohort stays open for later ones.
          const now = yield* DateTime.now;
   -      yield* disposeDelegatedCompletionCohort({
   -        command,
   -        events,
   -        projection,
   -        parentRunId: queuedMessage.delegatedCompletion.parentRunId,
   -        disposition: "disposed",
   -        now,
   -      });
   +      const carried = queuedMessage.delegatedCompletion.taskIds;
   +      const emitEvent = emit(events, command);
   +      for (const task of projection.subagents) {
   +        if (!carried.includes(task.id) || task.completionDelivery?.state !== "claimed") continue;
   +        yield* emitEvent({
   +          type: "subagent.updated",
   +          threadId: queuedRun.threadId,
   +          ...(task.runId === null ? {} : { runId: task.runId }),
   +          nodeId: task.id,
   +          driver: task.driver,
   +          providerInstanceId: task.providerInstanceId,
   +          occurredAt: now,
   +          payload: {
   +            ...task,
   +            completionDelivery: { state: "disposed", observedByRunId: null },
   +            updatedAt: now,
   +          },
   +        });
   +      }
   +      yield* emitQueuedRunCancellation({ command, events, projection, run: queuedRun, now });
          return;
        }
        const queuedRootNode =

   With it:

   [repro] after queued-run.cancel: {"wakeStatus":"cancelled","incoming":"disposed","siblingStatus":"running","siblingDelivery":null,"cohort":"open"}
   [repro] after sibling completes: {"successor":{"generation":2,"messageId":"message:thread:sibling-queued-cancel-parent:ordinal:2:…","taskIds":["node:delegated-task:sibling-queued-cancel-sibling"]}}
   [repro] late task after it completes: {"lateTaskDelivery":"claimed","cohort":"open"}
   Tests  2 passed | 39 skipped (41)
   

   DelegatedCompletionDelivery, ThreadStop, runtimeLayer, GuestShare, ThreadDeletion, SubagentProjection and ThreadManagementService tests: 245 passed, 0 failed. effect-tsgo tsrs --noEmit in apps/server reports no errors in Orchestrator.ts (only the existing brand suggestions at :200-:332, which are outside the patch).

   One call for you: thread.settle also goes through dispatchQueuedRunCancel for automatic queued runs (:3529). With this patch, settling no longer disposes those cohorts (the settle tests still pass). If settle should keep dropping later results, keep the old disposeDelegatedCompletionCohort call on that path only. thread.archive is not affected; it disposes all cohorts separately (:4317).

   Test body after the shared seeding (same as :1383, minus the write that completes the origin run)

   yield* orchestrator.dispatch({
     type: "message.dispatch", createdBy: "agent", creationSource: "server",
     commandId: CommandId.make(`${key}-wake`), threadId, messageId: delivery.messageId,
     text: "Delegated task completed", attachments: [], modelSelection,
     dispatchMode: { type: "queue_after_active" },
     delegatedCompletion: { parentRunId: originRunId, generation: delivery.generation, taskIds: delivery.taskIds },
   });
   yield* orchestrator.resumeQueuedRuns;
   const wake = (yield* orchestrator.getThreadProjection(threadId)).runs.find(
     (run) => run.userMessageId === delivery.messageId,
   )!;
   assert.equal(wake.status, "queued"); // origin is still running
   
   const cancelled = yield* orchestrator.dispatch({
     type: "queued-run.cancel", commandId: CommandId.make(`${key}-cancel`), threadId, runId: wake.id,
   });
   const cancelDelivery = new Map(cancelled.storedEvents.flatMap(({ event }) =>
     event.type === "subagent.updated" ? [[event.payload.id, event.payload.completionDelivery?.state] as const] : []));
   const afterCancel = yield* orchestrator.getThreadProjection(threadId);
   assert.equal(afterCancel.runs.find((run) => run.id === wake.id)?.status, "cancelled");
   assert.equal(cancelDelivery.get(incomingTaskId), "disposed");
   assert.notEqual(
     afterCancel.subagents.find((task) => task.id === sibling.id)?.completionDelivery?.state,
     "disposed",
     "a still-running sibling that was not in the cancelled message must keep its wake",
   );
   // then complete B's child run (as completeSibling does at :1490), wait for B's
   // subagent.updated with completionDelivery "claimed", and assert the cohort's
   // delivery is a new message whose taskIds are [sibling.id].

   Not run: a real provider turn. This is the event-level test only.
3. juliusmarminge

   closed this as completedin #32815on Nov 10, 2026