Skip to content

[remote-copilot-sdk] End-to-end integration test: SOURCE AgentChat + remote CopilotSdkChatClient + split tool routing + persistence round-trip #1319

Description

@JoshuaRowePhantom

Part of #1313

Summary

Add the end-to-end integration test (the acceptance criterion of #1313) that stands up a SOURCE AgentChat whose IChatClient is a ChatClientOverTransport reaching an in-process REMOTE instance where a CopilotSdkChatClient is built by ChatClientTransportListener from the wire-carried AgentDefinition, driven by the hermetic scripted Copilot SDK harness (S4). Exercise one user turn that (i) invokes a source-session tool locally, (ii) exercises the SDK's built-in shell tool remotely under the SDK session node, and (iii) round-trips history + CopilotSdkSessionId through the SOURCE's InMemoryAgentPersistenceStore.

Dependencies

Root Cause / Current State

  • features/Phantom.Workspaces.Transport.Tests/Scenarios/ByokTransportScenarioTests.cs hard-codes "provider": "github-models" (not github-copilot) in the AgentDefinition JSON (:24); CreateByokExecutor (:29-41) queues one streaming text update on a DeterministicTestChatClient and returns a bare IChatClient — no AgentChat, no tool registration, no persistence, no assertion about tool routing.
  • Combined with the (previously-open) server-side gap addressed by [remote-copilot-sdk] Complete + host server-side chat-client transport listener (builds CopilotSdkChatClient from AgentDefinition) #1314, the transport layer has zero coverage of the Copilot-SDK-BYOK + AgentChat + tools + persistence path.
  • features/Phantom.Workspaces.Llm.Core.Tests/CopilotByokTests.cs:109-137 is the only end-to-end Copilot-SDK-BYOK path today, and it is opt-in via COPILOT_BYOK_E2E — not a CI acceptance test for [remote-copilot-sdk] End-to-end test missing: Copilot-SDK-BYOK AgentChat with source-session tool + built-in PowerShell tool + persistence round-trip #1313's scenario.
  • features/Phantom.Workspaces.Transport.Tests/Infrastructure/HubRelayHarness.cs provides the in-process loopback needed to stand up the REMOTE side.
  • Persistence resume wiring: features/Phantom.Workspaces.Llm.Core/AgentChat.cs:373 (SetResumeSessionId), :374/:378 (SetCopilotSdkSessionId), :377 (SessionEstablished subscription); features/Phantom.Workspaces.Llm.Core/IncrementalPersistenceChatHistoryProvider.cs:48/:107/:143 (CopilotSdkSessionId writes).

Affected Files

  • features/Phantom.Workspaces.Transport.Tests/Scenarios/RemoteCopilotSdkSessionTests.cs (new) — the three tests below.
  • features/Phantom.Workspaces.Transport.Tests/Infrastructure/ — small additions to reuse HubRelayHarness for a REMOTE profile hosting CopilotSdkChatClient (per S1) driven by the scripted harness (per S4).
  • No production code changes here — this sub-item is test-only. All production changes required for the scenario land in S1–S5.

Design / Fix

  1. Harness assembly. Using HubRelayHarness, host an in-process REMOTE user-computer-profile that:
    • Registers ChatClientTransportListener (per S1) so the chat-client channel builds a CopilotSdkChatClient from the wire-carried AgentDefinition.
    • Injects the S4 scripted client factory (ICopilotClientFactory / Func<..., ICopilotClient>) so the SDK is driven from a queue.
    • Provides a REMOTE-only marker observable by the SOURCE side (e.g. a spy shell/PowerShell process factory or captured working directory) so tests can verify remote execution.
  2. SOURCE AgentChat. Construct on the source with:
    • IChatClient = ChatClientOverTransport via TransportTrustedExecutor.CreateAgentChatAsync under the RouterLocalChatClientRemote ExecutorTopology (per S2 + S3).
    • Real InMemoryAgentPersistenceStore.
    • A source-session tool registered (workspace-gui / workspace-entity or an agent-session/current-session tool targeting the source AgentSessionId).
  3. Scripted remote turn. The scripted session emits:
    • SessionEstablished with a stable id.
    • Assistant text deltas.
    • A tool call for the registered source-session tool.
    • A root-AgentId ToolExecutionStartEvent + ToolExecutionCompleteEvent for the built-in shell/powershell tool (per S5).
    • Tool result for the source-side tool.
    • Session idle.
  4. Assertions — the three named tests below.
  5. No ambient network. No real copilot.exe.

Expected Tests

The three tests below ARE the acceptance criteria for #1313 (names per #1313's Desired Test section):

Test Asserts
RemoteCopilotSdkSession_SourceSessionToolInvokedOnSourceInstance Given a SOURCE AgentChat whose IChatClient is a ChatClientOverTransport reaching a REMOTE in-process instance running CopilotSdkChatClient (built by the extended ChatClientTransportListener from the wire-carried AgentDefinition), and a source-targeted tool (workspace-gui / workspace-entity / agent-session targeting the source) registered on the SOURCE, a scripted SDK tool call for that tool is dispatched by the source's router to LocalTrustedExecutor (verified via spy) and executes against the SOURCE session's AgentServices.CurrentSessionContext / IRunningAgentChatFactory, NOT on the remote instance. The FunctionResultContent is delivered back into the same turn on the SOURCE.
RemoteCopilotSdkSession_BuiltinPowerShellToolRunsOnRemoteUnderSessionNode Given the same setup, a scripted ToolExecutionStartEvent / ToolExecutionCompleteEvent pair with root AgentId == null for a built-in shell/powershell tool executes on the REMOTE instance (verified by a REMOTE-only marker, e.g. spy on the remote shell/PowerShell factory or a captured remote working directory), and the resulting transcript items on the SOURCE are written to the root/session sink (no CopilotSdkStreamAdapter.ParentToolCallIdPropertyName), appear as children of the SDK session node in AgentChat history, and are not dropped by the adapter.
RemoteCopilotSdkSession_HistoryAndPersistenceRoundTripOnSource After the turn, closing and re-opening the SOURCE AgentChat against the SOURCE's InMemoryAgentPersistenceStore restores the full transcript: user message, assistant text, BOTH tool-call and tool-result items (source-local tool AND remote built-in tool) with ParentToolCallId tags intact. On restore, CopilotSdkChatClient.SetResumeSessionId (per AgentChat.cs:373) is invoked on the resumed REMOTE client with the SDK session id established during the original turn, and PersistedAgent.CopilotSdkSessionId matches.

Acceptance = all three pass with a hermetic loopback transport (HubRelayHarness or equivalent) and a scripted remote Copilot SDK session; no ambient network, no real copilot.exe.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingdiagnosedRoot cause identifiedverified-locallyImplementation has been verified locally

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions