Release gate flake: RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource drops final tool-result because dispose cancels the persistence drain (blocks v0.0.21)

[JoshuaRowePhantom]

Blocks v0.0.21

This failure is release-blocking for v0.0.21. The v0.0.21 tag was moved to the exact verified tip f04c8524bccf8aab60762806e3a1ccb7474e77c9 ("Fix Copilot validator native exit state", 2026-09-16 09:06:56 -0700), but the release workflow run 35132787733 failed at the Run test suite (release gate) step, so Publish and package win-x64 and Create GitHub Release never ran. No GitHub Release or assets exist for v0.0.21. A single test failed (7,425 passed / 1 failed / 7,426 executed).

Related history (do not reopen): #1313 (epic — end-to-end Copilot-SDK-BYOK persistence round-trip) and #1319 (the sub-item that authored this exact test) are closed. This is a new, distinct defect: the test's contract is correct, but it has a synchronization race against incremental persistence that #1319 did not cover. #1577 is a separate, unrelated intermittent AgentChatResumeTests NRE. This issue references them for lineage only.

---

Summary

Phantom.Workspaces.Transport.Tests.Scenarios.RemoteCopilotSdkSessionTests.RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource intermittently fails at the persistence read-back assertion for the last message of turn 1 (FunctionResultContent with CallId == "call-shell-1"). The persisted collection is [first, turn-one, "", ""] — exactly the user message, the assistant message, the source tool-result, and the shell tool-call — with the final shell tool-result message missing.

The root cause is a genuine ordering race between two facts that are both by-design:

1. Persistence of a turn's final message lags live history by exactly one update. StreamingPersistenceMiddleware persists a message only once it is "stable" — i.e. once the next update begins, or once an update carrying FinishReason arrives. The CopilotSdkStreamAdapter emits that terminal FinishReason=Stop update on SessionIdleEvent specifically so the last message becomes durable. So a message appears in live chat.History one update before it is persisted.
2. Disposal cancels the process loop before it drains that terminal update. AgentChat.DisposeCoreAsync calls await this.cts.CancelAsync() before awaiting processTask, and cts.Token is the token threaded into the streaming pull inside the middleware.

The test synchronizes on live chat1.History reaching 5 (WaitForHistoryCountAsync, line 192) and then immediately closes chat1 (end of the await using block). If the process loop has not yet pulled + persisted the terminal FinishReason update by the time cancellation fires, the last message (call-shell-1 tool-result) is never written, and the subsequent SourceStore.ReadMessagesAsync (line 203) returns only 4 messages. Under CI thread-pool contention (the middleware offloads each pull via Task.Run) plus the added in-process wire latency of the transport topology, the disposal-cancellation wins the race — which is why it reproduces only under the release-CI environment/order and not in the local exact-tip full gate (7,658/7,658).

Classification: primarily a test/harness synchronization defect (the assertion races persistence it cannot observe yet), which also exposes a latent product data-loss window — disposing an AgentChat immediately after a turn's last message appears can drop that message from persisted history. Both the test-first fix and the product hardening are described below.

---

Observed vs Expected

Observed (release gate, run 35132787733):

FAIL: Phantom.Workspaces.Transport.Tests.Scenarios.RemoteCopilotSdkSessionTests.RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource
Assert.Contains() Failure: filter not matched in collection
Collection (persisted messages read back from SourceStore): [first, turn-one, "", ""]

The failing assertion is line 208:

Assert.Contains(messages, m => m.Contents.OfType<FunctionResultContent>().Any(r => r.CallId == "call-shell-1"));

The four persisted messages are: user "first", assistant "turn-one"+FunctionCallContent(call-source-1), tool FunctionResultContent(call-source-1) (""), assistant FunctionCallContent(call-shell-1) (""). The fifth message — tool FunctionResultContent(call-shell-1) — is missing. (Lines 205–207 pass because those contents are in the first four messages; line 208 is the first assertion touching content that lives only in the un-persisted fifth message.)

Expected: All of turn 1's messages, including the final FunctionResultContent(call-shell-1), are durably persisted to the source InMemoryAgentPersistenceStore before the round-trip is asserted, so the read-back contains the complete transcript and (line 227–230) the reopened chat2.History restores it in full.

---

Reproduction

1. Run the fast suite (scripts/run-tests.ps1 -Mode fast) under load / high parallelism (release CI or a loaded dev box), or add artificial thread-pool starvation.
2. The turn-1 event script is: assistant delta turn-one → ToolExecutionStart/Complete(call-source-1) → ToolExecutionStart/Complete(call-shell-1) → SessionIdle.
3. WaitForHistoryCountAsync(chat1.History, 5, ...) (line 192) returns as soon as the call-shell-1 result update is yielded into live history — which is before the middleware pulls the subsequent terminal FinishReason update that would persist it.
4. The await using (var chat1 …) block closes → DisposeAsync → cts.CancelAsync() (AgentChat.cs:1746) fires before the process loop drains the terminal update.
5. SourceStore.ReadMessagesAsync (line 203) then returns [first, turn-one, "", ""] and line 208 fails.

Deterministic local repro (for the fix author): after WaitForHistoryCountAsync(chat1.History, 5, …), inject a Task.Delay/yield-starvation before the terminal update is pulled, or dispose chat1 on a starved scheduler, and the final tool-result is reliably dropped.

---

Evidence / Log excerpt

Release gate step Run test suite (release gate) (run 35132787733, job 104917619366):

=== Test Run Summary ===
Executed : 7426
Passed   : 7425
Failed   : 1
FAIL [runneradmin_runnervmvmocb_2026-09-16_18_46_45_net10.0]: 1 test(s) failed:
  Phantom.Workspaces.Transport.Tests.Scenarios.RemoteCopilotSdkSessionTests.RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource
##[error]Process completed with exit code 1.

Downstream Publish and package win-x64 and Create GitHub Release were skipped; earlier steps (Build, version derive, SDK pin assert, MXC native unit) all passed. Earlier local exact-tip full release gate passed 7,658/7,658, confirming the failure is non-deterministic and environment/order-sensitive rather than a hard regression.

---

Root Cause

The persisted collection [first, turn-one, "", ""] is missing exactly one message: the final tool-result. That is the fingerprint of "the turn's last message reached live history but was never persisted." Three code facts combine to make this happen on immediate dispose:

1. The middleware persists the last message only on a terminal FinishReason update.
features/Phantom.Workspaces.Llm.Core/StreamingPersistenceMiddleware.cs:93-100:

var response = buffer.ToChatResponse();
var stableCount = update.FinishReason is not null
    ? response.Messages.Count          // final: persist ALL messages (incl. the last)
    : Math.Max(0, response.Messages.Count - 1);  // in-flight: last message is NOT yet stable
for (var i = persistedCount; i < stableCount; i++)
    await this.PersistMessageAsync(response.Messages[i]).ConfigureAwait(false);

When the call-shell-1 result update arrives, FinishReason is still null, so stableCount = messagesCount - 1 and the call-shell-1 result message (the last one) is not persisted. It is yielded to the consumer (advancing live History to 5) but only becomes durable when the next update — the terminal one — is pulled.

2. The adapter's terminal FinishReason update is what makes the last message durable.
features/Phantom.Workspaces.Llm.Core/CopilotSdkStreamAdapter.cs:263-274 (on SessionIdleEvent):

// Emit a terminal update carrying FinishReason so downstream middleware
// (StreamingPersistenceMiddleware) treats the response as final and persists
// the last message of the turn. Without this the last message of every
// Copilot turn is treated as unstable and never persisted (issue #1103).
yield return new ChatResponseUpdate { Role = ChatRole.Assistant, FinishReason = ChatFinishReason.Stop };
yield break;

This update must be pulled across the transport (remote ChatClientTransportListener → wire → source ChatClientOverTransport → StreamingPersistenceMiddleware) for call-shell-1's result to be written. IncrementalPersistenceChatHistoryProvider.StoreChatHistoryAsync is a deliberate no-op — the middleware is the sole writer of response messages — so if the terminal update is not drained, nothing else persists the last message.

3. Disposal cancels the drain.
features/Phantom.Workspaces.Llm.Core/AgentChat.cs:1746-1767 (DisposeCoreAsync):

await this.cts.CancelAsync();   // cancels the process-loop token FIRST
...
try { await this.processTask; } catch (OperationCanceledException) { }

processTask runs RunProcessLoopAsync(this.cts.Token) (AgentChat.cs:2847-2851), and that same cts.Token flows into the streaming pipeline, including the middleware's Task.Run(() => enumerator.MoveNextAsync(), cancellationToken). Cancelling the CTS before the loop drains the terminal update aborts the pull, so call-shell-1's result is never persisted.

The race window: WaitForHistoryCountAsync(chat1.History, 5, …) (test line 192) returns the instant the call-shell-1 result update is yielded — i.e. after fact (1), before facts (2)/(3) complete. The test then disposes chat1 (fact 3's cancellation) which can beat the drain (fact 2). Under load + wire latency the cancellation wins → missing final message.

---

Affected Files

File	Role in the defect
features/Phantom.Workspaces.Transport.Tests/Scenarios/RemoteCopilotSdkSessionTests.cs	The flaky test. Synchronizes on live chat1.History count (line 192) then disposes chat1 and reads persistence (lines 203–208), racing the one-update-late persistence of the final message. Primary fix site.
features/Phantom.Workspaces.Llm.Core/StreamingPersistenceMiddleware.cs	stableCount logic (lines 93–100): the last message is persisted only on a terminal FinishReason update, so persistence legitimately lags live history by one update.
features/Phantom.Workspaces.Llm.Core/CopilotSdkStreamAdapter.cs	Emits the terminal FinishReason=Stop update on SessionIdleEvent (lines 263–274) that the middleware needs to persist the last message.
features/Phantom.Workspaces.Llm.Core/AgentChat.cs	DisposeCoreAsync cancels this.cts (line 1746) before awaiting processTask (line 1767); cts.Token drives the process loop (2847–2851) and the streaming pull. Product-hardening site.
features/Phantom.Workspaces.Llm.Core/IncrementalPersistenceChatHistoryProvider.cs	StoreChatHistoryAsync is a no-op (sole-writer contract), so nothing backstops the middleware if its final drain is cancelled.

---

Design / Fix

Option A — Test-first synchronization fix (primary, lowest risk)

Make the test wait for persistence to reflect the turn's final message, rather than for live history. Add a store-polling helper and use it before disposing chat1 / before the read-back asserts:

// After WaitForHistoryCountAsync(chat1.History, 5, ...):
await WaitForPersistedAsync(
    () => setup.SourceStore.ReadMessagesAsync(new ReadMessagesRequest { AgentSessionId = chat1.AgentSessionId }, ct),
    msgs => msgs.Any(m => m.Contents.OfType<FunctionResultContent>().Any(r => r.CallId == "call-shell-1")),
    "source store to persist the final call-shell-1 tool-result",
    timeoutSeconds: 60);

WaitForPersistedAsync polls the store (same shape as the existing WaitForConditionAsync, lines 391–410) until the final tool-result is durable, then the await using close and read-back are deterministic. This removes the flake without asserting anything the product does not already guarantee once the turn fully drains.

Option B — Product hardening (recommended in addition; closes the latent data-loss window)

Guarantee that disposal drains the in-flight (already-complete) turn's persistence before cancelling. In AgentChat.DisposeCoreAsync, give the current turn a bounded grace period to reach natural stream completion before cts.CancelAsync() (e.g. await processTask with a timeout on a graceful-shutdown token, then hard-cancel). This ensures that disposing a chat immediately after a turn's last message appears in History does not drop that message from persisted history in production, not just in tests.

Considered / background: persisting the last buffered message from the middleware's finally block on cancellation was rejected — on a genuine mid-token cancellation the last buffered message is not yet complete, so flushing it could persist a half-streamed message. The terminal-FinishReason signal is the only reliable "last message is complete" marker, which is why the fix belongs at the drain/lifecycle level (Option B) and/or the test-sync level (Option A), not in an unconditional cancel-time flush.

Recommended resolution: ship A to unblock v0.0.21 deterministically, and land B to eliminate the underlying data-loss window.

---

Expected Tests

Test Name	Class	What It Verifies
RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource	RemoteCopilotSdkSessionTests	Hardened to poll SourceStore.ReadMessagesAsync for FunctionResultContent(call-shell-1) before closing chat1 and asserting, so the final tool-result round-trips deterministically (no dependence on drain-vs-dispose timing).
AgentChat_DisposeImmediatelyAfterTurnCompletes_PersistsFinalTurnMessage	AgentChatPersistenceIntegrationTests	Disposing an AgentChat immediately after a turn's last message appears in live History still persists that final message (drains the terminal FinishReason update before cancelling) — asserts Option B at the product level.
StreamingPersistenceMiddleware_LastMessageStabilizedByTerminalFinishReason_IsPersistedBeforeStreamCompletes	StreamingPersistenceMiddlewareTests	With a scripted stream ending in a terminal FinishReason update, the final message is persisted; and if the consumer stops pulling before the terminal update, the final message is (documented as) not yet durable — pinning the one-update-late contract the race exploits.

---

Acceptance Criteria

• RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource passes deterministically, including under thread-pool starvation / high parallelism (e.g. 50+ consecutive and stress runs green).
• The read-back at line 203–208 and the reopened chat2.History restore (lines 226–230) both contain the complete turn-1 transcript, including FunctionResultContent(call-shell-1).
• (Option B) Disposing an AgentChat immediately after a turn completes does not drop the turn's final message from persisted history, proven by AgentChat_DisposeImmediatelyAfterTurnCompletes_PersistsFinalTurnMessage.
• The v0.0.21 release gate (scripts/run-tests.ps1 -Mode fast) passes with 0 failures on the exact tip so Publish and package win-x64 and Create GitHub Release can run.
• No change to the persistence sole-writer contract (IncrementalPersistenceChatHistoryProvider.StoreChatHistoryAsync stays a no-op); no unconditional cancel-time flush of unstable messages.

---

Filed by the file-bug role. Failure source: release workflow run 35132787733 on tip f04c8524bccf8aab60762806e3a1ccb7474e77c9. Lineage: #1313, #1319 (closed; not reopened), #1577 (unrelated intermittent NRE).

[JoshuaRowePhantom]

Design

Root cause

StreamingPersistenceMiddleware deliberately persists only stable messages. The final FunctionResultContent(call-shell-1) is visible in live history before the following FinishReason=Stop update makes it durable. AgentChat.DisposeCoreAsync currently cancels the process token first, so immediate disposal can cancel that terminal pull and permanently lose the final message.

Approach

• Add an AgentChat-owned active-turn completion barrier. Disposal requests the process loop to stop after the current turn, waits for that barrier through a short bounded grace period driven by the injected TimeProvider, then performs the existing hard cancellation. A naturally completing terminal update therefore persists and projects before cancellation; a provider that never emits terminal still gets cancelled when the bound expires.
• Strengthen the remote round-trip test by awaiting the existing AgentInputTurnCompletion.Settlement lifecycle signal rather than polling live history or storage.
• Add deterministic gated tests proving immediate disposal waits for and persists a terminal tool result, and proving the timeout path still cancels a non-terminating provider without sleeps.
• Pin middleware ordering directly: the last tool result is not durable before the terminal update is pulled, and is durable before that terminal update is yielded.

Alternatives considered

• Polling the store or adding sleeps only hides the race and is rejected.
• Unconditionally flushing on cancellation could persist a partial streamed message and is rejected.
• Waiting forever for idle/terminal would regress cancellation responsiveness and is rejected.

Files

• Phantom.Workspaces.Llm.Core/AgentChat.cs
• Phantom.Workspaces.Llm.Core.Tests/AgentChatPersistenceIntegrationTests.cs
• Phantom.Workspaces.Llm.Core.Tests/AgentChatDisposalTests.cs
• Phantom.Workspaces.Llm.Core.Tests/StreamingPersistenceMiddlewareTests.cs
• Phantom.Workspaces.Transport.Tests/Scenarios/RemoteCopilotSdkSessionTests.cs

[JoshuaRowePhantom]

Implementation notes

Commits: 96e2417352e79ea27e928f3c0f8be4cad4dd6502, d03a41b9dd165041af1f68546b644d1c91bd75ef, 6fcd91eba85167b1c3a441720223d990a217e155, dcf7b5c06c2e9b4ed85c6c1db16860cecda2bf5b

Deviations from proposed design

After the check-in diagnosis, the bounded drain now uses a dedicated internal disposal TimeProvider rather than the domain/history clock. It defaults to TimeProvider.System for real elapsed-time liveness and is separately injectable only through the internal creation contract for deterministic tests.

Scope changes

Added directly coupled child-chat propagation and explicit hosted-stream cancellation coverage so hosted sub-agents retain and prove the same lifecycle policy without inheriting frozen domain time. No other lifecycle clocks required correction.

Files changed

• Phantom.Workspaces.Llm.Core/AgentChat.cs
• Phantom.Workspaces.Llm.Core/InternalCreateAgentChatRequest.cs
• Phantom.Workspaces.Llm.Core.Tests/AgentChatDisposalTests.cs
• Phantom.Workspaces.Llm.Core.Tests/AgentChatPersistenceIntegrationTests.cs
• Phantom.Workspaces.Llm.Core.Tests/StreamingPersistenceMiddlewareTests.cs
• Phantom.Workspaces.Transport.Tests/Scenarios/RemoteCopilotSdkSessionTests.cs

Correction verification

• scripts/build.ps1: succeeded with 0 warnings and 0 errors.
• Targeted suite, including AgentViewModelSubAgentTreeFilterTests, AgentChatHostedSubAgentTests, and explicit frozen-domain hosted cancellation coverage: 72/72 passed.
• Final repository stability script: all 10 runs passed (72/72 each run).
• No dump files produced.

[JoshuaRowePhantom]

Verification result

Status: ✅ Verified

Commits checked (features..HEAD, base f04c8524, HEAD d03a41b9):

d03a41b9 Test #1583: make disposal ordering deterministic
96e24173 Fix #1583: drain terminal persistence before disposal

Root-contract assessment (independent inspection — fix notes not trusted)

• Terminal update cannot be dropped by immediate dispose. AgentChat.DisposeCoreAsync now waits on an activeTurnCompletion barrier before cts.CancelAsync(). The barrier completes in the process loop's inner finally only after the streaming loop has fully drained — i.e. after the terminal FinishReason=Stop update is pulled through StreamingPersistenceMiddleware (which persists the last message). So disposal after live history shows the final tool-result cannot cancel the persistence drain.
• Deterministic barrier, no sleeps/polling/timing. Barrier = await activeTurn.WaitAsync(DisposeDrainTimeout, this.timeProvider) using the injected TimeProvider. Tests drive it via FakeTimeProvider.Advance (timeout path) and gated MarkReady() / WaitForClaimedAsync() (drain path). No Task.Delay/Thread.Sleep/polling in new code.
• Cancellation, provider-failure, no-deadlock, bounded. DisposeDrainTimeout = 2s bounds the wait; on TimeoutException it hard-cancels. Dispose_WhileProcessLoopWaitingForResponse proves a never-terminating provider still disposes (disposal blocks → advance time → completes). Provider errors keep the existing diagnostic path; activeTurnCompletion.TrySetResult() in the inner finally guarantees the barrier never hangs on error/cleanup exceptions.
• No duplicate persistence / reorder / swallowed errors / zombie tasks. Middleware persistence and the sole-writer contract are untouched (IncrementalPersistenceChatHistoryProvider.StoreChatHistoryAsync still => ValueTask.CompletedTask, not in diff). Barrier completes exactly once (ReferenceEquals-guarded null-out); only TimeoutException is caught; turn faults still surface via await this.processTask. RunContinuationsAsynchronously avoids inline-continuation deadlock.
• Mode compatibility & concurrency. Both RunProcessLoopAsync (remote/source) and RunHostedProcessLoopAsync (hosted sub-agents) set+complete the barrier. A Volatile.Read(disposeStarted) guard on the loop condition prevents starting a new turn after dispose begins; per-turn replacement is lock-guarded; nested try/finally guarantees completion across cleanup paths.

Adversarial tests (force old ordering; assert exact final result)

• AgentChat_DisposeImmediatelyAfterTurnCompletes_PersistsFinalTurnMessage — gates the terminal update, reproduces the exact race (final call-shell-1 tool-result in live History (count 3) but unpersisted), disposes, asserts disposal blocks (Assert.False(disposal.IsCompleted)), releases, then asserts the exact persisted FunctionResultContent(call-shell-1) (not a count). Would fail on base (base cancels the terminal pull → final message dropped).
• StreamingPersistenceMiddleware_LastMessageStabilizedByTerminalFinishReason_IsPersistedBeforeStreamCompletes — pins the one-update-late contract with exact call-shell-1 result assertions (absent before terminal pull, present after) via a gated TerminalGatingChatClient.
• RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource — synchronization changed from racing live-history-count to awaiting turn.Settlement (completes only after drain+persistence); retains exact call-shell-1 FunctionResultContent read-back and chat2.History full-transcript restore assertions.

Outstanding gaps

None — all criteria satisfied. (Pre-existing timing patterns at RemoteCopilotSdkSessionTests.cs lines 326/407/417 are outside the 4-line diff and explicitly left unchanged by the issue design; out of scope.)

Tests

Targeted classes: RemoteCopilotSdkSessionTests, AgentChatPersistenceIntegrationTests, AgentChatDisposalTests, StreamingPersistenceMiddlewareTests — built with 0 warnings/0 errors.

=== Test Run Summary ===
Executed : 30
Passed   : 30
Failed   : 0
  Passed! - Failed: 0, Passed: 27, Total: 27 - Phantom.Workspaces.Llm.Core.Tests.dll (net10.0)
  Passed! - Failed: 0, Passed:  3, Total:  3 - Phantom.Workspaces.Transport.Tests.dll (net10.0)

Stability: run-tests-stability.ps1 — 10/10 consecutive runs passed (30/30 each). No hang dumps (.dmp) produced. Working tree clean; HEAD d03a41b9.

Slow suite not required (diff touches no filesystem/Git repository layers).

[JoshuaRowePhantom]

Diagnosis: test failure is related to this change

Failing test: Phantom.Workspaces.Agent.Gui.Tests.AgentViewModelSubAgentTreeFilterTests.SubAgentsTree_LatestCompletionBumpsToTopOfCompletedGroup

Gate: scripts/run-tests.ps1 -Mode fast — 6,944 passed, 0 assertion failures, but the Agent GUI test host timed out and produced hang dumps.

Causal link: this branch changes AgentChat.DisposeCoreAsync to await activeTurnCompletion with WaitAsync(DisposeDrainTimeout, this.timeProvider) before cancellation. The failing pre-existing test injects a frozen FakeTimeProvider, creates two hosted sub-agents whose RunHostedProcessLoopAsync streams remain blocked on ChannelReader.ReadAllAsync, and then disposes the view model. Because the fake clock is no longer advanced, the new two-second drain timeout never expires and disposal hangs.

The dump confirms the test awaiting AgentViewModel.DisposeAsync → nested AgentChat.DisposeAsync/DisposeCoreAsync, while both hosted loops await SubAgentChatClient.GetStreamingResponseAsync / channel reads. This is directly exercised by the modified disposal path, so checkin stopped. No fast-forward, push, close, label change, amend, rebase, or sentinel reset was performed.

[JoshuaRowePhantom]

Design correction after check-in diagnosis

The gate diagnosis is correct: AgentChat.TimeProvider is domain time (history timestamps/order and deterministic domain behavior), so it cannot own a lifecycle safety timeout. A frozen domain FakeTimeProvider can otherwise make disposal unbounded.

Correction

• Split disposal timeout scheduling from domain time in the internal creation contract. The disposal clock defaults to TimeProvider.System, whose timers advance with real elapsed time in production, independent of wall-clock/domain overrides.
• Flow that dedicated lifecycle clock to child chats; never flow the parent's domain clock into disposal timeout scheduling.
• Keep the active-turn barrier and ordering unchanged: normally completing terminal updates still drain through incremental persistence before cancellation. If terminal never arrives, only the dedicated bounded lifecycle clock releases disposal to hard-cancel.
• Deterministically inject a separate FakeTimeProvider only for timeout-path tests. Keep the domain fake frozen, advance only the lifecycle fake, and assert cancellation/disposal completes. In the normal terminal test, freeze both clocks, release the terminal event, and assert the exact final FunctionResultContent(call-shell-1) persisted before disposal completes.
• Run the original four targeted classes, AgentViewModelSubAgentTreeFilterTests, and AgentChatHostedSubAgentTests, then 10-run targeted stability.

Coupled timeout audit

AgentChat.DisposeCoreAsync is the only injected-domain-time use in AgentChat that schedules a lifecycle safety timeout. Other nearby TimeProvider uses stamp/order domain state and remain unchanged.

Files

• Phantom.Workspaces.Llm.Core/InternalCreateAgentChatRequest.cs
• Phantom.Workspaces.Llm.Core/AgentChat.cs
• Phantom.Workspaces.Llm.Core.Tests/AgentChatDisposalTests.cs
• Phantom.Workspaces.Llm.Core.Tests/AgentChatPersistenceIntegrationTests.cs

[JoshuaRowePhantom]

Verification result

Status: ✅ Verified

Fresh, reasoning-isolated verify-locally retry at HEAD dcf7b5c06c2e9b4ed85c6c1db16860cecda2bf5b (branch fix/1583-drain-persistence-before-dispose), base features = f04c8524bccf8aab60762806e3a1ccb7474e77c9. This supersedes the earlier zero-gap comment (checked d03a41b9, before the check-in hang) — it independently re-verifies the design correction that followed the AgentViewModelSubAgentTreeFilterTests gate hang. Fix notes were not trusted; every claim was re-derived from the code and tests.

Commits checked (features..HEAD):

dcf7b5c0 Test #1583: cover hosted disposal timeout
6fcd91eb Fix #1583: isolate disposal timeout from domain time
d03a41b9 Test #1583: make disposal ordering deterministic
96e24173 Fix #1583: drain terminal persistence before disposal

Product diff limited to AgentChat.cs and InternalCreateAgentChatRequest.cs; the rest are tests.

Root-contract 1 — immediate dispose drains persistence before cancel

• DisposeCoreAsync now awaits an activeTurnCompletion barrier before cts.CancelAsync() (AgentChat.cs:1758–1766). The barrier is completed in the process loop's inner finally (2379–2387), which runs only after the streaming await-loop has fully drained — i.e. after the terminal FinishReason=Stop update is pulled through StreamingPersistenceMiddleware, which persists the last message. So disposing immediately after live History shows the final call-shell-1 tool-result cannot cancel that persistence drain.
• Exact final message, no duplicate/reorder. Sole-writer contract untouched — IncrementalPersistenceChatHistoryProvider.StoreChatHistoryAsync => ValueTask.CompletedTask (not in diff). AgentChat_DisposeImmediatelyAfterTurnCompletes_PersistsFinalTurnMessage asserts the exact FunctionResultContent(call-shell-1) with Result == "shell-ok" via Assert.Single (proves no duplicate write), not a count.
• No swallowed errors / unobserved tasks / hangs. Only TimeoutException is caught around the barrier wait; turn faults still surface via await this.processTask. The barrier TrySetResult() sits in a nested finally, so it completes even if cleanup throws; RunContinuationsAsynchronously avoids inline-continuation deadlock; the barrier is the only new awaitable and disposal awaits it.

Root-contract 2 — lifecycle clock independent of frozen domain time

• New dedicated disposalTimeProvider; InternalCreateAgentChatRequest.DisposalTimeProvider defaults to TimeProvider.System (real monotonic elapsed time). The bounded wait uses await activeTurn.WaitAsync(DisposeDrainTimeout /*2s*/, this.disposalTimeProvider) — never the domain TimeProvider callers may freeze.
• Cannot be controlled by untrusted callers / cannot leak across chats. DisposalTimeProvider lives only on the internal creation record; the public factory path (AgentFactory.cs:641) wires only the domain TimeProvider and leaves DisposalTimeProvider at its System default (AgentChatFactory.cs sets neither). The only non-default propagation is parent→child at AgentChat.cs:1499, which forwards the dedicated disposal clock (System in production), never the domain clock.
• Cannot hang hosted/sub-agent disposal. RunHostedProcessLoopAsync sets+completes the barrier; a hosted stream blocked on a channel read is released by the System-clock 2s bound then hard-cancel. Verified by AgentChat_HostedDispose_WithFrozenDomainTime_CancelsWhenDisposalTimeoutExpires (asserts disposal blocks, advancing the injected disposal fake completes it, and CancellationObserved == true) and by the previously-hanging GUI class.
• Normal terminal completion wins without cancellation. The immediate-dispose integration test freezes both the domain and disposal fakes and never advances them — releasing the gated terminal lets the barrier complete naturally, so disposal finishes with no timeout/cancel.

Specific scenarios re-checked

• Previously hanging AgentViewModelSubAgentTreeFilterTests.SubAgentsTree_LatestCompletionBumpsToTopOfCompletedGroup — reproduced the exact setup (frozen domain FakeTimeProvider + two hosted sub-agents disposed via the view model). Now green; the whole GUI class passes and no .dmp is produced.
• Hosted stream disposal, frozen-time adversarial timeout, remote source round-trip (turn.Settlement replaces the racy live-History count; Settlement resolves only after providerCleanup, i.e. after the terminal update is drained+persisted; retains exact call-shell-1 read-back and chat2.History restore), one-update-late middleware contract (StreamingPersistenceMiddleware_LastMessageStabilizedByTerminalFinishReason_IsPersistedBeforeStreamCompletes), and multiple-turn / error-path coverage (TwoTurns_StoreContainsBothTurns, NUpdates_EachPersistedExactlyOnce, TurnCancelled..., CancellationOfConsumer_DoesNotAbortInFlightPersistence, DisposeAsync_OwnedResourceFailure...) all pass.
• No forbidden patterns in changed/new code: no Task.Delay/Thread.Sleep/polling/timing assertions, no dispatcher-pass pumping, no .Result/.Wait()/.GetAwaiter().GetResult(), no [Skip]/quarantine, no unbacked TODO. Determinism uses TaskCompletionSource gating + FakeTimeProvider.Advance. New tests use the existing internal InternalCreateAgentChatRequest/EnqueueUserMessageWithCompletion/DisposeDrainTimeout seams via friend assemblies — no public API misuse. (The pre-existing Task.Delay poller at RemoteCopilotSdkSessionTests.cs:396–435 is the [remote-copilot-sdk] End-to-end integration test: SOURCE AgentChat + remote CopilotSdkChatClient + split tool routing + persistence round-trip #1319 second-turn resume check, outside this 4-line diff — out of scope.)

Outstanding gaps

None — all criteria and both root contracts satisfied.

Tests

Combined targeted run via scripts\run-tests.ps1 -Mode fast (build: 0 warnings / 0 errors):

=== Test Run Summary ===
Executed : 72
Passed   : 72
Failed   : 0
  Passed! - Failed: 0, Passed: 14, Total: 14 - Phantom.Workspaces.Agent.Gui.Tests.dll (net10.0)
  Passed! - Failed: 0, Passed: 55, Total: 55 - Phantom.Workspaces.Llm.Core.Tests.dll (net10.0)
  Passed! - Failed: 0, Passed:  3, Total:  3 - Phantom.Workspaces.Transport.Tests.dll (net10.0)

Classes: RemoteCopilotSdkSessionTests, AgentChatPersistenceIntegrationTests, AgentChatDisposalTests, StreamingPersistenceMiddlewareTests, AgentViewModelSubAgentTreeFilterTests (the previously-hanging GUI class), AgentChatHostedSubAgentTests.

Stability: run-tests-stability.ps1 — 10/10 consecutive runs passed (72/72 each), including the GUI class. No hang dumps (.dmp) produced across any run. Working tree clean; HEAD dcf7b5c0.

Slow suite not required — the diff touches no filesystem or Git repository layers (only Phantom.Workspaces.Llm.Core + test projects).