Skip to content

Re-connecting should be part of DCP request retry - #18121

Merged
Karol Zadora-Przylecki (karolz-ms) merged 1 commit into
mainfrom
dev/karolz/retry-connection-main
Jun 11, 2026
Merged

Karol Zadora-Przylecki (karolz-ms) merged 1 commit into
mainfrom
dev/karolz/retry-connection-main

Conversation

@karolz-ms

Copy link
Copy Markdown
Contributor

Manual backport (forward-port? 😄) of #18096 into main branch

Copilot AI review requested due to automatic review settings June 11, 2026 18:06
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18121

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18121"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR is a manual forward-port of PR #18096 into main. It moves the DCP API server connection establishment (EnsureKubernetesAsync) inside the retry loop of ExecuteWithRetry, so that failures caused by a missing or partially-written kubeconfig file are properly retried. This addresses timing issues where DCP hasn't finished writing its kubeconfig when Aspire first tries to read it (issues #18050 and #18041).

Changes:

  • Moved EnsureKubernetesAsync from before the retry pipeline into the resiliencePipeline.ExecuteAsync lambda, allowing kubeconfig read failures to be retried by the outer resilience pipeline. Removed the now-redundant dedicated timeout pipeline wrapper inside EnsureKubernetesAsync and switched to explicit FileStream with FileShare.ReadWrite | FileShare.Delete for non-exclusive kubeconfig reads.
  • Added a new test class KubernetesServiceTests with two integration tests verifying the retry behavior when the kubeconfig is initially missing or partially written, using a lightweight fake DCP API server backed by Kestrel on a random port.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/Aspire.Hosting/Dcp/KubernetesService.cs Moves EnsureKubernetesAsync inside the retry loop in ExecuteWithRetry; removes the inner timeout pipeline from EnsureKubernetesAsync; opens the kubeconfig file with FileShare.ReadWrite | FileShare.Delete to avoid interfering with DCP writes.
tests/Aspire.Hosting.Tests/Dcp/KubernetesServiceTests.cs New test file with two tests for the retry-on-kubeconfig-unavailable behavior, along with test helpers (TestFileSystemService, TestDcpApiServer).

@karolz-ms

Copy link
Copy Markdown
Contributor Author

PR Testing Report

PR Information

Artifact Version Verification

  • Expected Commit: 3c195e9
  • Tested Artifact: Source checkout at PR head commit (no "Dogfood this PR" CLI artifact — the PR is a draft with no comments, so there is no published CLI to dogfood)
  • Installed Version: N/A — built and tested the source checkout at the PR head commit
  • Status: ✅ Verified (HEAD == PR head commit)

Changes Analyzed

Files Changed

  • src/Aspire.Hosting/Dcp/KubernetesService.cs — Modified (+42 / -35)
  • tests/Aspire.Hosting.Tests/Dcp/KubernetesServiceTests.cs — Added (+287)

Change Categories

  • Hosting core changes (DCP KubernetesService retry/connection logic)
  • Test changes (new KubernetesServiceTests)
  • CLI changes
  • Dashboard changes
  • Template changes
  • Client/Component changes
  • CI infrastructure changes
  • VS Code extension changes

What the change does

  • Moves DCP connection establishment (EnsureKubernetesAsync) inside the resilience retry loop in ExecuteWithRetry, so re-connecting (e.g. due to a missing or partially-written kubeconfig) becomes part of the request retry rather than a one-time step before the loop.
  • Reads the kubeconfig via a FileStream opened with FileShare.ReadWrite | FileShare.Delete so the host does not interfere with DCP if it opens the file mid-write.
  • Restructures the read/timeout resilience pipeline (drops the separate timeout-wrapping pipeline; the read pipeline now drives the file-wait + build directly).

Test Scenarios Executed

Scenario 1: Check out PR branch + build

Objective: Confirm the PR source restores and builds cleanly.
Coverage Type: Build validation
Status: ✅ Passed

Steps:

  1. git fetch origin pull/18121/head and checked out head commit 3c195e9d5.
  2. Ran restore.cmd to set up the local .NET SDK.
  3. Built Aspire.Hosting.Tests (which builds Aspire.Hosting) via dotnet test.

Observations:

  • Restore succeeded: 0 warnings, 0 errors.
  • Test project + dependencies compiled with 0 warnings, 0 errors.

Scenario 2: Run the 2 new KubernetesServiceTests

Objective: Directly validate the fix — connection is established/re-established inside the retry loop.
Coverage Type: Happy path + boundary (missing / partially-written kubeconfig)
Status: ✅ Passed (2/2)

Steps:

  1. dotnet test --project tests/Aspire.Hosting.Tests/... -- --filter-class "*.KubernetesServiceTests" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"

Tests run:

  • ExecuteWithRetry_EstablishesConnection_WhenKubeconfigInitiallyMissing ✅ — kubeconfig absent at call time; the operation waits and succeeds once DCP writes it.
  • ExecuteWithRetry_EstablishesConnection_WhenKubeconfigInitiallyPartiallyWritten ✅ — kubeconfig exists but is a partial/invalid prefix; the read is retried and succeeds once the complete file is appended.

Observations:

  • Both tests passed (total 2, succeeded 2, failed 0) in ~6.8s. These exercise exactly the reconnect-in-retry behavior the PR introduces.

Scenario 3: DCP namespace regression sweep

Objective: Ensure the KubernetesService refactor didn't regress existing DCP behavior.
Coverage Type: Regression
Status: ✅ Passed (222/222)

Steps:

  1. dotnet test --project tests/Aspire.Hosting.Tests/... --no-build -- --filter-namespace "Aspire.Hosting.Tests.Dcp" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"

Observations:

  • All 222 tests in Aspire.Hosting.Tests.Dcp passed (failed 0, skipped 0) in ~31s.
  • Covers DcpExecutorTests, DcpCliArgsTests, DcpHostNotificationTests, DcpLogParserTests, ConfigureDefaultDcpOptionsTests, ProcessUtilTests, ResourceSnapshotBuilderTests, and the new KubernetesServiceTests.

Notes

  • The test project (Aspire.Hosting.Tests) is single-targeted to $(DefaultTargetFramework) = net8.0 (per eng/Versions.props), so net8.0 is the only configured TFM; there is no additional TFM to run. The change is framework-agnostic C#.
  • No end-to-end aspire run smoke test was performed (user opted for scenarios 1–3). The reconnect-on-missing/partial-kubeconfig path is precisely covered by the new unit tests, and normal DCP connect is covered by the broader DCP suite.

Summary

Scenario Status Notes
1. Build PR source ✅ Passed Restore + build: 0 warnings, 0 errors
2. New KubernetesServiceTests ✅ Passed 2/2
3. DCP regression sweep ✅ Passed 222/222

Overall Result

✅ PR VERIFIED

The new tests directly validate the reconnect-inside-retry behavior, and the full DCP test namespace passes with no regressions. Build is clean.

Recommendations

  • None blocking. Optionally, a manual aspire run smoke test against a real DCP could be done before merge for extra confidence, but it is largely redundant with the unit + regression coverage for this specific change.

@github-actions

Copy link
Copy Markdown
Contributor

Re-running the failed jobs in the CI workflow for this pull request because 3 jobs were identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

  • Tests / Hosting.Keycloak / Hosting.Keycloak (windows-latest) - Failed step 'Build test project' will be retried because the job log shows a likely transient infrastructure network failure. Matched pattern: /Unable to load the service index for source https:\/\/(?:pkgs\.dev\.azure\.com\/dnceng|dnceng\.pkgs\.visualstudio\.com)\/public\/_packaging\//i.
  • Tests / MySqlConnector / MySqlConnector (ubuntu-latest) - Failed step 'Build test project' will be retried because the job log shows a likely transient infrastructure network failure. Matched pattern: /Unable to load the service index for source https:\/\/(?:pkgs\.dev\.azure\.com\/dnceng|dnceng\.pkgs\.visualstudio\.com)\/public\/_packaging\//i.
  • Tests / Hosting-1 / Hosting-1 (windows-latest) - Job-level runner or infrastructure failure matched the transient allowlist.

@github-actions

Copy link
Copy Markdown
Contributor

❌ CLI E2E Tests failed — 114 passed, 1 failed, 2 unknown (commit 3c195e9)

❌ Failed Tests

- Test Detail
❌ AllPublishMethodsBuildDockerImages Recording · Job · CLI logs
View all recordings
- Test Detail
✅ AddPackageInteractiveWhileAppHostRunningDetached Recording · Job · CLI logs
✅ AddPackageWhileAppHostRunningDetached Recording · Job · CLI logs
✅ AgentCommands_AllHelpOutputs_AreCorrect Recording · Job · CLI logs
✅ AgentInitCommand_DefaultSelection_InstallsDefaultSkills Recording · Job · CLI logs
✅ AgentInitCommand_MigratesDeprecatedConfig Recording · Job · CLI logs
✅ AgentInit_NonInteractive_BundleOnlySkillsNotInCatalog Recording · Job · CLI logs
✅ AgentMcpListResources_ExcludesResourceMarkedWithExcludeFromMcp Recording · Job · CLI logs
✅ AgentMcpListStructuredLogsReturnsLogsFromStarterApp Recording · Job · CLI logs
✅ AgentMcpListStructuredLogsReturnsLogsFromStarterApp_DevLocalhost Recording · Job · CLI logs
✅ AgentMcpListStructuredLogsReturnsLogsFromStarterApp_Isolated Recording · Job · CLI logs
❌ AllPublishMethodsBuildDockerImages Recording · Job · CLI logs
✅ AspireAddAndStartWorkAgainstLegacyAppHostTs Recording · Job · CLI logs
✅ AspireAddPackageVersionToDirectoryPackagesProps Recording · Job · CLI logs
✅ AspireInitSingleFileAppHostRunsViaDotnetRunAppHost Recording · Job · CLI logs
✅ AspireInit_ExistingAppHostDir_RecreatesNuGetConfigKeepsFiles Recording · Job · CLI logs
❔ AspireInit_SolutionFile_BuildsAgainstChannelHive Recording · Job · CLI logs
✅ AspireStartUpdatesStaleTypeScriptAppHostPath Recording · Job · CLI logs
✅ AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps Recording · Job · CLI logs
✅ AspireUpdateRemovesOrphanAppHostPackageVersionWhenSdkAlreadyCurrent Recording · Job · CLI logs
✅ Banner_DisplayedOnFirstRun Recording · Job · CLI logs
✅ Banner_DisplayedWithExplicitFlag Recording · Job · CLI logs
✅ Banner_NotDisplayedWithNoLogoFlag Recording · Job · CLI logs
✅ CertificatesClean_RemovesCertificates Recording · Job · CLI logs
✅ CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate Recording · Job · CLI logs
✅ CertificatesTrust_WithUntrustedCert_TrustsCertificate Recording · Job · CLI logs
✅ ConfigSetGet_CreatesNestedJsonFormat Recording · Job · CLI logs
✅ CreateAndRunAspireStarterProject Recording · Job · CLI logs
✅ CreateAndRunAspireStarterProjectWithBundle Recording · Job · CLI logs
✅ CreateAndRunEmptyAppHostProject Recording · Job · CLI logs
✅ CreateAndRunJavaEmptyAppHostProject Recording · Job · CLI logs
✅ CreateAndRunJsReactProject Recording · Job · CLI logs
✅ CreateAndRunPolyglotAppHostWithDevLocalhostUrls Recording · Job · CLI logs
✅ CreateAndRunPythonReactProject Recording · Job · CLI logs
✅ CreateAndRunTypeScriptEmptyAppHostProject Recording · Job · CLI logs
✅ CreateAndRunTypeScriptStarterProject Recording · Job · CLI logs
✅ CreateJavaAppHostWithViteApp Recording · Job · CLI logs
✅ CreateTypeScriptAppHostWithViteApp_UsesConfiguredToolchain Recording · Job · CLI logs
✅ DashboardRunWithAgentMcpListTracesReturnsNoTraces Recording · Job · CLI logs
✅ DashboardRunWithAgentMcpListTracesReturnsNoTraces_DevLocalhost Recording · Job · CLI logs
✅ DashboardRunWithOtelTracesReturnsNoTraces Recording · Job · CLI logs
✅ DashboardRunWithOtelTracesReturnsNoTraces_DevLocalhost Recording · Job · CLI logs
✅ DeployK8sBasicApiService Recording · Job · CLI logs
✅ DeployK8sWithExternalHelmChart Recording · Job · CLI logs
✅ DeployK8sWithGarnet Recording · Job · CLI logs
✅ DeployK8sWithMongoDB Recording · Job · CLI logs
✅ DeployK8sWithMySql Recording · Job · CLI logs
✅ DeployK8sWithPostgres Recording · Job · CLI logs
✅ DeployK8sWithRabbitMQ Recording · Job · CLI logs
✅ DeployK8sWithRedis Recording · Job · CLI logs
✅ DeployK8sWithSqlServer Recording · Job · CLI logs
✅ DeployK8sWithValkey Recording · Job · CLI logs
✅ DeployTypeScriptAppToKubernetes Recording · Job · CLI logs
✅ DescribeCommandResolvesReplicaNames Recording · Job · CLI logs
✅ DescribeCommandShowsRunningResources Recording · Job · CLI logs
✅ DetachFormatJsonProducesValidJson Recording · Job · CLI logs
✅ DetachFormatJsonProducesValidJsonWhenRestartingExistingInstance Recording · Job · CLI logs
✅ DoPublishAndDeployListStepsWork Recording · Job · CLI logs
✅ DocsCommand_RendersInteractiveMarkdownFromLocalSource Recording · Job · CLI logs
✅ DoctorCommand_DetectsDeprecatedAgentConfig Recording · Job · CLI logs
✅ DoctorCommand_TypeScriptAppHostReportsMissingConfiguredToolchain Recording · Job · CLI logs
✅ DoctorCommand_WithSslCertDir_ShowsTrusted Recording · Job · CLI logs
✅ DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted Recording · Job · CLI logs
✅ DotNetRunFileBasedAppHostUsesAspireCliBundle Recording · Job · CLI logs
✅ DotNetRunProjectAppHostUsesAspireCliBundle Recording · Job · CLI logs
✅ GatewayWithoutExternalEndpoint_FailsPublishWithGuidance Recording · Job · CLI logs
✅ GeneratedAspireDevScript_StartsWatchMode_WithConfiguredToolchain Recording · Job · CLI logs
✅ GlobalMigration_HandlesCommentsAndTrailingCommas Recording · Job · CLI logs
✅ GlobalMigration_HandlesMalformedLegacyJson Recording · Job · CLI logs
✅ GlobalMigration_PreservesAllValueTypes Recording · Job · CLI logs
✅ GlobalMigration_SkipsWhenNewConfigExists Recording · Job · CLI logs
✅ GlobalSettings_MigratedFromLegacyFormat Recording · Job · CLI logs
✅ IngressWithoutExternalEndpoint_FailsPublishWithGuidance Recording · Job · CLI logs
✅ InitTypeScriptAppHost_AugmentsExistingViteRepoInWorkspaceSubdirectory Recording · Job · CLI logs
✅ InteractiveCSharpInitCreatesExpectedFiles Recording · Job · CLI logs
✅ InvalidAppHostPathWithComments_IsHealedOnRun Recording · Job · CLI logs
✅ JavaScriptHostingApisRunFromTypeScriptAppHost Recording · Job · CLI logs
✅ LatestCliCanStartStableChannelAppHost Recording · Job · CLI logs
✅ LatestCliCanStartStableChannelTypeScriptAppHost Recording · Job · CLI logs
✅ LegacySettingsMigration_AdjustsRelativeAppHostPath Recording · Job · CLI logs
✅ LogsCommandShowsResourceLogs Recording · Job · CLI logs
✅ OtelLogsReturnsStructuredLogsFromStarterApp Recording · Job · CLI logs
✅ OtelLogsReturnsStructuredLogsFromStarterAppIsolated Recording · Job · CLI logs
✅ ProcessCommandCallbackReceivesCliArguments Recording · Job · CLI logs
✅ PsCommandListsRunningAppHost Recording · Job · CLI logs
✅ PsFormatJsonOutputsOnlyJsonToStdout Recording · Job · CLI logs
✅ PublishJavaScriptPatternsGeneratesExpectedDockerComposeArtifacts Recording · Job · CLI logs
✅ PublishWithConfigureEnvFileUpdatesEnvOutput Recording · Job · CLI logs
✅ PublishWithDockerComposeServiceCallbackSucceeds Recording · Job · CLI logs
✅ PublishWithoutOutputPathUsesAppHostDirectoryDefault Recording · Job · CLI logs
✅ ResourceCommand_FailedExec_ShowsLogPathAndLogHasEntries Recording · Job · CLI logs
✅ ResourceCommand_SetAndDeleteParameterUpdatesDescribeOutput Recording · Job · CLI logs
✅ RestoreGeneratesSdkFiles Recording · Job · CLI logs
✅ RestoreGeneratesSdkFiles_WithConfiguredToolchain Recording · Job · CLI logs
✅ RestoreRefreshesGeneratedSdkAfterAddingIntegration Recording · Job · CLI logs
✅ RestoreSupportsConfigOnlyHelperPackageAndCrossPackageTypes Recording · Job · CLI logs
✅ RunFromParentDirectory_UsesExistingConfigNearAppHost Recording · Job · CLI logs
✅ RunReportsSyntaxErrorsForDotNetAppHost Recording · Job · CLI logs
✅ RunReportsSyntaxErrorsForTypeScriptAppHost Recording · Job · CLI logs
✅ SecretCrudOnDotNetAppHost Recording · Job · CLI logs
✅ SecretCrudOnTypeScriptAppHost Recording · Job · CLI logs
✅ StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels Recording · Job · CLI logs
✅ StartAndWaitForTypeScriptSqlServerAppHostWithNativeAssets Recording · Job · CLI logs
✅ StartReportsSyntaxErrorsForDotNetAppHost Recording · Job · CLI logs
✅ StartReportsSyntaxErrorsForTypeScriptAppHost Recording · Job · CLI logs
✅ StopAllAppHostsFromAppHostDirectory Recording · Job · CLI logs
✅ StopJavaPolyglotAppHostUsingApphostDirectory Recording · Job · CLI logs
✅ StopNonInteractiveSingleAppHost Recording · Job · CLI logs
✅ StopTypeScriptPolyglotAppHostUsingApphostDirectory Recording · Job · CLI logs
✅ StopWithNoRunningAppHostExitsSuccessfully Recording · Job · CLI logs
✅ TerminalAttachFrontend_ShowsViteHelpAndDetaches Recording · Job · CLI logs
✅ TypeScriptAppHostRunDoesNotDeadlockWhenLazyOptionsInvokeAsyncCallback Recording · Job · CLI logs
✅ TypeScriptAppHostWithVite_AllowsDifferentGuestPkgManager Recording · Job · CLI logs
✅ UnAwaitedChainsCompileWithAutoResolvePromises Recording · Job · CLI logs
❔ UpdateToStable_CSharpEmptyAppHost_KeepsConfigChannel Recording · Job · CLI logs
✅ UpdateToStable_CSharpSingleFileInit_KeepsConfigChannel Recording · Job · CLI logs
✅ UpdateToStable_TypeScriptSingleFileInit_KeepsConfigChannel Recording · Job · CLI logs
✅ UpdateToStable_TypeScript_PreviewsStablePkgsAndKeepsChannel Recording · Job · CLI logs

📹 Recordings uploaded automatically from CI run #27367459470

@karolz-ms
Karol Zadora-Przylecki (karolz-ms) merged commit 2d41a8f into main Jun 11, 2026
661 of 671 checks passed
@karolz-ms
Karol Zadora-Przylecki (karolz-ms) deleted the dev/karolz/retry-connection-main branch June 11, 2026 21:06
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 13.5 milestone Jun 11, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

docs_optional → bug_fix_restores_documented_behavior

No documentation update is needed for this PR.

Triggered signals: none (signal_count == 0)

Allowlist justification: The two changed files are:

  • src/Aspire.Hosting/Dcp/KubernetesService.cs — moves EnsureKubernetesAsync inside the resilience retry loop so that a missing or partially-written DCP kubeconfig is retried automatically; removes an inner dedicated timeout pipeline; opens the kubeconfig with FileShare.ReadWrite | FileShare.Delete to avoid interfering with concurrent DCP writes.
  • tests/Aspire.Hosting.Tests/Dcp/KubernetesServiceTests.cs — new test file covering the two retry scenarios.

No new public types, methods, options, configuration keys, CLI flags, resource strings, container image tags, or user-visible default values were introduced. The PR fixes issues #18050 and #18041 where Aspire could fail to connect to DCP if it tried to read the kubeconfig before DCP finished writing it. The intended behavior — that Aspire connects to DCP successfully on startup — was already the documented, expected behavior; this change makes the implementation match that expectation more reliably under timing pressure.

@github-actions github-actions Bot locked and limited conversation to collaborators Jul 12, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants