[CI Postmortem] Windows bot VSM-XAM-126: 'Dotnet tests' consistently failing (path not found)

[rolfbjarne]

Summary

The Windows CI bot VSM-XAM-126 has a 44% job failure rate (8/18) across 7 unrelated PRs this week. All failures in the 'Dotnet tests' job produce:

Path does not exist: c:\AzDO\_work\8\s\macios\tests\TestSummary.md

This suggests the test run on this bot fails before producing results, likely due to a misconfigured or stale workspace.

Affected PRs

#25079, #25105, #25204, #25220, #25227, #25229, #25235

Classification

• Type: Infrastructure — bot-specific
• Impact: 8 failed jobs across 7 PRs in one week
• Confidence: High — 100% of 'Dotnet tests' failures on this bot have the same path error; other bots (VSM-XAM-133, 135, 141) show similar but less concentrated patterns

Other Windows Bots

Bot	Failed/Total	Rate	PRs
VSM-XAM-126	8/18	44%	25079, 25105, 25204, 25220, 25227, 25229, 25235
VSM-XAM-141	3/6	50%	25079, 25195, 25238
VSM-XAM-133	3/10	30%	25079, 25195, 25229
VSM-XAM-135	3/9	33%	25204, 25231

Other Windows bots (133, 135, 141) also have elevated failure rates with similar Path does not exist errors (different drive letters: C:, D:).

---

Filed automatically by CI postmortem analysis.

[rolfbjarne]

Not enough diagnostic information here to be actionable.

[rolfbjarne]

CI Post-Mortem Update (Apr 27 – May 4, 2026)

The 'Path does not exist' error is still occurring and is now broader than just VSM-XAM-126.

This Week's Data

• 78 occurrences across 11 PRs, 16 builds
• Affected bots include: VSM-XAM-126, VSM-XAM-141, VSM-XAM-13.Tahoe.arm64, VSM-XAM-20.Tahoe.arm64, VSM-XAM-31.Monterey.arm64, VSM-XAM-56.Sequoia.arm64, VSM-XAM-58.Tahoe.arm64, VSM-XAM-70.Ventura.arm64, XAMBOT-1006.Sonoma, XAMBOT-1025.Sonoma, VSCXSDKs-MINI-016/024/033/049

Sample Builds

PR	Build	Bot
#25261	13954991	VSM-XAM-126
#25197	13955255	VSM-XAM-141
#25267	13955396	VSM-XAM-13.Tahoe.arm64
#25279	13958912	Multiple bots
#25239	13980194	VSM-XAM-20.Tahoe.arm64

This is no longer isolated to one Windows bot — it's appearing across macOS bots as well.

---

This update was automatically generated by CI post-mortem analysis.

[rolfbjarne]

Updated: Specific Job/Step Links

Here are direct links to the exact failing steps (click to see the error in context):

Windows bots (Dotnet tests → Publish Artifact: TestSummary)

PR	Build	Bot	Direct Link
#25261	13954991	VSM-XAM-126	Dotnet tests → Publish TestSummary
#25197	13955255	VSM-XAM-141	Dotnet tests → Publish TestSummary
#25269	13955798	VSM-XAM-135	Dotnet tests → Publish TestSummary
#25279	13958912	VSM-XAM-141	Dotnet tests → Publish TestSummary

Error: Path does not exist: c:\AzDO\_work\N\s\macios\tests\TestSummary.md

macOS bots (macOS tests → Publish Artifact: HtmlReport / TestSummary)

PR	Build	Bot	Direct Link
#25267	13955396	VSM-XAM-13.Tahoe.arm64	macOS tests → Publish HtmlReport
#24889	13955500	VSM-XAM-13.Tahoe.arm64	macOS tests → Publish TestSummary
#25239	13955657	VSM-XAM-13.Tahoe.arm64	macOS tests → Publish HtmlReport
#24889	13958843	XAMBOT-1006.Sonoma	macOS tests → Publish HtmlReport
#25279	13958912	VSM-XAM-31.Monterey.arm64	macOS tests → Publish HtmlReport

Error: Path does not exist: /Users/builder/azdo/_work/N/a/HtmlReport.zip

Pattern

The missing files are always test artifacts (TestSummary.md or HtmlReport.zip), meaning the test run itself likely failed/crashed before producing these outputs. This happens on both Windows and macOS bots, across many different machines.

[rolfbjarne]

Correction: 'Path does not exist' is a symptom, not the root cause

The Path does not exist errors on Publish Artifact: TestSummary/HtmlReport steps have continueOnError: true — they are warnings, not the actual failures. The test artifacts don't exist because an earlier step failed before tests could run.

Actual root causes in those jobs:

Failing Task	Occurrences
Install dotnet workloads	22
Checkout (git clone/merge)	7
Download secrets: azdev-secrets	4
Prepare tests results and Html Report	4
Provisionator dependencies	3
Verify ssh connection	2
Other (Download Mac tests, Agent logs, etc.)	3

The dominant failure is Install dotnet workloads (22 of 45 cases). The 'Path does not exist' on artifact publish is just the downstream consequence of these earlier failures.

This issue should be retitled/refocused on the actual infrastructure failures, or split into separate issues per root cause.

[rolfbjarne]

Closing in favor of split issues by root cause:

• [CI Postmortem] 'Install dotnet workloads' failures causing cascading test failures #25312 — Install dotnet workloads failures (22 occurrences, 2 PRs)
• [CI Postmortem] 'Verify ssh connection' failing on Windows 'Dotnet tests' job #25313 — Prepare tests results and Html Report on Windows bots (4 occurrences, 4 PRs)
• [CI Postmortem] 'Download secrets: azdev-secrets' failing on VSM-XAM-13 #25314 — Download secrets: azdev-secrets on VSM-XAM-13 (4 occurrences, 3 PRs)

Remaining items were single-PR or too few occurrences to track separately:

• Checkout/git fetch failures (1 PR each)
• Provisionator dependencies (1 PR)
• Verify ssh connection (2 PRs, 2 occurrences)
• Download Mac tests timeout (1 PR, 1 occurrence)