[CI] Provisionator Xcode downloads at ~160 KB/s hit the 180-minute job limit

[dalexsoto]

Additional info to collect

Copilot: please collect the following information for each occurrence of this issue:

• The name of the bot (to see if this happens only on specific bots)
• The IP address of the bot (to see if this happens only on specific subnets).

Failure signature

Provisionator dependencies continuously downloads Xcode at roughly 160 KB/s until Azure DevOps cancels the job at its 180-minute limit. The logs show normal progress rather than a source/build error, but the download cannot finish before the hard cap.

Confirmed occurrences

PR / head	Build and job	Observed progress
#26144 / 1fd611914a973a3efd935e92aae627a9579664da	14717583 - cecil	Xcode 26.6: 1% at 10:35:35 UTC to 73% at 13:25:26 UTC
#26144 / 1fd611914a973a3efd935e92aae627a9579664da	14717583 - monotouch_macos	Xcode 26.6: 1% at 10:45:46 UTC to 74% at 13:36:32 UTC
#26144 / 1fd611914a973a3efd935e92aae627a9579664da	14717583 - interdependent-binding-projects	Xcode 26.6: 1% at 10:35:38 UTC to 76% at 13:30:33 UTC
#26180 / 897e85a4b8e8e206c57ea59a7b0c3088abf18e15	14718322 - monotouch_ios	Xcode 27.0 beta 3: 1% at 11:36:49 UTC to 89% at 14:32:05 UTC

First observed: 2026-07-21 10:35 UTC. Latest observed: 2026-07-21 14:32 UTC. Recurrence: 4 jobs across 2 distinct monitored PRs.

Impact and diagnosis

The affected jobs never reach their tests, publish no usable test summary, and block otherwise unrelated PRs. Sibling jobs on the same builds complete successfully, and the only terminal error is Azure DevOps reporting that the job ran longer than the maximum time of 180 minutes; task tails show the Xcode download still making slow progress at cancellation. This is a CI infrastructure/network-throughput failure, not a source regression.

Workaround

Rerun failed jobs after download bandwidth recovers, subject to the normal retry and merge-conflict gates. Artifact and task-log retention is limited, so the linked logs may eventually expire.

[dalexsoto]

New recurrence on PR #26200 @ 966f7cc95666adf0926c09964990868e5f161864, build 14718622:

• cecil: Xcode 27 beta 3 downloaded at ~154 KB/s, reaching only 88% before the 180-minute cap.
• linker_ios: the same download reached only 87% before the cap.

The signature now affects 6 jobs across 3 monitored PRs. Classification remains CI infrastructure; no retry has been requested for #26200 while the build is still in progress. Retained task logs may expire.

[dalexsoto]

New affected PR: #26202 @ 1f6d35df981f18fd18fb67e70d4def97495f1daa, build 14718710, job dotnettests_ios. Xcode 27 beta 3 downloaded at ~158 KB/s and reached only 83% before Azure canceled the job at 180 minutes.

The signature now affects 7 jobs across 4 monitored PRs. No retry has been requested while build 14718710 remains in progress; retained logs may expire.

[dalexsoto]

Additional slow-download cancellations:

• PR [msbuild] Generate dSYM files by default on macOS/Mac Catalyst when using Native AOT. Fixes #21553 #26197 @ e7db69490124cd20737e92fb2189a460cd845c46: build 14718443, job sharpie, Xcode 26.6 reached 75% at ~161 KB/s before the cap.
• PR [CI] Make 'Clean for remote tests' able to delete read-only directories #26198 @ 0a8a95dd2f117727a1b51b46e043def81ca7664a: build 14718483, job linker_maccatalyst, Xcode 26.6 reached 74% at ~162 KB/s.
• Already-listed PR [MediaToolbox] Update bindings up to Xcode 27.0 Beta 3 #26202 added dotnettests_maccatalyst and linker_macos, reaching 89% and 88% respectively before the cap.

The signature now affects 11 jobs across 6 monitored PRs. PR #26198 already has an equivalent automatic same-head run, build 14720520, in progress, so no duplicate retry was submitted. PR #26197 has a deterministic source-related failure in the same stage; PR #26202 remains in progress.

[dalexsoto]

New affected PR: #26204 @ e91862693a70d34dfd66bcc1d6967fe986fb464b, build 14718779, job Build macOS tests. Provision Products & Frameworks downloaded Xcode 27 beta 3 at ~162 KB/s and reached only 90% before Azure canceled the job at 180 minutes.

The signature now affects 12 jobs across 7 monitored PRs. Build 14718779 remains in progress, so no retry has been requested.

[dalexsoto]

PR #26204 build 14718779 added two more jobs with the same signature: fsharp and sharpie, both reaching only 90% of Xcode 27 beta 3 at ~165 KB/s before the 180-minute cap. The signature now affects 14 jobs across 7 monitored PRs.

Retry status: PR #26200 build 14718622 simulator_tests attempt 2 was accepted. PR #26204 build 14718779 simulator_tests attempt 2 and build_macos_tests attempt 2 were accepted as two separately counted requests; all remain queued.

[dalexsoto]

New affected PR: #26203 @ abd0ea0058d6285dccbdcc9c0fc4837d264b45d0, build 14718863, job dotnettests_macos. Xcode 27 beta 3 downloaded at ~163 KB/s and reached only 88% before the 180-minute cap.

The signature now affects 15 jobs across 8 monitored PRs. Build 14718863 remains in progress, so no retry has been requested.

[dalexsoto]

New affected PR: #26158 @ eca5c5b88045ef1ed57d5e572cbea1f4fd2a9cde, build 14719100, job generator. Xcode 27 beta 3 downloaded at ~159 KB/s and reached only 88% before the 180-minute cap.

The signature now affects 16 jobs across 9 monitored PRs. Build 14719100 remains in progress, so no retry has been requested.

[dalexsoto]

Already-listed PR #26207 added linker_macos: Xcode 27 beta 3 downloaded at ~164 KB/s and reached only 90% before the 180-minute cap.

The signature now affects 17 jobs across 9 monitored PRs. Build 14719170 remains in progress, so no retry has been requested.

[dalexsoto]

Two additional monitored PRs hit the same slow Xcode 26.6 download cap:

• PR [msbuild] Copy dSYMs from xcframeworks next to the app's dSYM. Fixes #23076. #25979 @ bf96da28d2420b4b949ae80f3b8a50caec3b032b: build 14720177, job dotnettests_maccatalyst, reaching only 75% at ~160 KB/s.
• PR [src] Clean/consolidate runtime initialization code between platforms. #26131 @ 610d74fba2896839287f55d5b78a08997acac6fc: build 14720171, job interdependent-binding-projects, reaching only 75% at ~163 KB/s.

The signature now affects 19 jobs across 11 monitored PRs. PR #26131 simulator_tests attempt 2 was accepted at 22:34 UTC; PR #25979 remains in progress.

[dalexsoto]

PR #25979 @ bf96da28d2420b4b949ae80f3b8a50caec3b032b, build 14720177, added a second affected job: monotouch_ios. Xcode 26.6 progressed from ~148 KB/s to ~160 KB/s and reached only 74% before the 180-minute cap.

The signature now affects 20 jobs across 11 monitored PRs. Build 14720177 is terminal; simulator_tests attempt 2 was accepted at 22:52 UTC and is pending. The reconciled retry count is 4 lifetime / 2 on this head.

[dalexsoto]

New affected PR: #26216 @ ddecc2fa8e582c067a9f27a07907f21618c276e1, build 14720305, job interdependent-binding-projects. Xcode 26.6 downloaded at ~160 KB/s and reached only 74% before the Azure 180-minute cap.

The signature now affects 21 jobs across 12 monitored PRs. Classification remains CI infrastructure. Build 14720305 completed failed at 23:06 UTC; no retry was submitted because the same simulator stage also contains four deterministic linker failures. Retained task logs may expire.

[dalexsoto]

New affected PR: #26214 @ cbe4d8f45bb02ec0f286f978d8314cff0841b1c0, build 14720423, job linker_tvos. Xcode 26.6 downloaded at ~159 KB/s and reached only 75% before the Azure 180-minute cap.

The signature now affects 22 jobs across 13 monitored PRs. Build 14720423 completed failed. No retry was submitted because GitHub reports the PR as CONFLICTING / DIRTY; retained task logs may expire.

[dalexsoto]

Retry outcome for the previously recorded PR #26213 occurrence: build 14719548 simulator_tests attempt 2 completed failed at 00:04 UTC. The retried xcframework job never reached provisioning because checkout repeatedly failed with fatal: couldn't find remote ref refs/pull/26213/merge after the PR merged.

This does not change the slow-Xcode diagnosis. No further retry will be submitted because the merged PR's synthetic merge ref is no longer available.

[dalexsoto]

New affected PR: #26146 @ f447dd24780dc6c6e0274723f10648c932af6324, build 14720461. Both dotnettests_macos and linker_tvos downloaded Xcode 26.6 at ~162 KB/s and reached only 75% before the 180-minute cap.

The signature now affects 24 jobs across 14 monitored PRs. Build 14720461 remains active, so no retry has been submitted; retained task logs may expire.

[dalexsoto]

PR #26146 retry update: build 14720461 became terminal, and no equivalent same-head build remains active. Simulator_tests attempt 2 was accepted at 00:27 UTC to rerun the two slow-Xcode jobs together with the other transient failures. Reconciled budget: 2 lifetime retries / 1 on the current head.