Fix or disable flaky tests

[jankratochvilcz]

Tracking issue for flaky tests that need to be fixed or permanently disabled.

Flaky tests

• ToolTaskThatTimeoutAndRetry in ToolTask_Tests.cs — consistently flaky even after multiple stabilization attempts. Currently skipped on CI via Skip flaky ToolTaskThatTimeoutAndRetry test on CI #13668.

Context

These tests are unreliable in CI and cause spurious failures. Each should be investigated to determine if it can be stabilized or should be removed.

[jankratochvilcz]

Fresh evidence on ToolTaskThatTimeoutAndRetry from a 4-day telemetry harvest of dnceng-public PR CI (77 builds):

• 2 failures at the post-Stabilize 3 ToolTask_Tests flakes with diagnostics + timing fix #13830 bumped budgets (slowDelay=20000, Timeout=5000).
• Both failures landed on the same PR: dotnet/msbuild#13863, which is a pure Arcade dependency bump (only eng/ and global.json changed) — no code path that could regress the test.
• On the surfaced retry results the test passed in 3413ms (under the 5s Timeout), so this is the classic flaky-fast-path shape: ToolTask.Execute startup overhead occasionally spikes past the timeout on cold-start.

Just closed a follow-up PR (dotnet/msbuild#13846) that was going to shrink the budgets — data argues the opposite direction. The post-#13830 budget is the floor, not the ceiling.

Realistic next steps for this entry: either a deterministic-sync rewrite (probably needs decoupling from Stopwatch-timed flow) or unconditional disable. Not planning to drive either myself but flagging the data here.

[jankratochvilcz]

Took a stab at the deterministic rewrite for the ToolTaskThatTimeoutAndRetry entry: jankratochvilcz/msbuild#50 (draft on my fork for now -- happy to mirror upstream if reviewers like the shape).

Approach: the test was sharing one task.Timeout value across every attempt, which coupled the success-path attempts (no semantic reason to be wall-clock-bounded) to the timing-out attempt. The fix sets Timeout per attempt:

\\csharp
bool isTimingOutAttempt = timeoutOnFirstExecution && attempt == 1;
task.Timeout = isTimingOutAttempt
? timeoutMilliseconds // 2s -- only the attempt we expect to time out
: System.Threading.Timeout.Infinite;
\\

Plus tightening slowDelay from 20s to 10s so the timing-out attempt has 5x margin (ping ~10s vs timeout 2s) before it can race the timeout. Net effect:

• Success-path attempts have zero wall-clock dependency -- they cannot flake under any CI load.
• Timing-out attempt has 5x headroom, which is well past any observed cold-start spike.
• Test wall clock drops from ~7-9s to ~4-6s.

Local verification: 5 consecutive runs × 6 inline cases = 30/30 passing on net10.0|x64 and net48|x86.

If this approach looks reasonable I can open the same change against dotnet/msbuild and we could close out this entry of #13667. Otherwise happy to iterate on shape.

[jankratochvilcz]

Opened #13864 (draft) with the deterministic rewrite proposed above.

[jankratochvilcz]

Flake survey 2026-05-27

Compared dotnet-msbuild-public pipeline 75 across two windows using the flaky-test-survey skill:

Metric	Pre (May 14-21, 7d)	Post (May 21-27, 6d)	Delta
Completed builds	128	111	--
Failed test records	332	168	-49%
Failed records / build	2.59	1.51	-42%
Builds with >=1 fail	51 (39.8%)	42 (37.8%)	-2pp

Per-build flake noise roughly halved post-merge wave. The "builds with any failure" metric barely moved because one persistent top offender drags the headline up.

Targeted fixes (status of recent PRs)

Test	Fix PR	Pre fails	Post fails	Verdict
TaskHostFactory_Tests.TaskNodesDieAfterBuild	#13828	1	0	Eliminated
ToolTask_Tests.HandleExecutionErrorsWhenToolLogsError	#13760 + #13830	4	2	Improved (~halved)
ToolTask_Tests.ToolTaskThatTimeoutAndRetry	#13830 -> #13864	7	3	Improved (most window before #13864 merged 05-25)
ToolTask_Tests.OverrideStdOutImportanceToHigh	#13830	2	2	No change
ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat	#13830	19	24 (22%)	Regressed -> next-wave fix #13874
ToolTask_Tests.ErrorWhenTextSentToStandardError	(none yet)	1	4	Same family as above; #13874 should cover it
ProcessLoggingEventConcurrentWithShutdown_DoesNotThrow	#13745	0	0	Holding
FileLogger_Tests.DistributedFileLoggerParameters	#13704	2	1	Slight improvement

Next-wave PRs

• Fix ToolTask EOF-truncation flake: drain on every wait slice + 30s budget (#13734) #13874 (draft) -- structural fix for the EOF-pipe-truncation race called out in ToolTask drops final stdout/stderr lines after Wave18_6 EOF wait — flaky ToolTaskCanChangeCanonicalErrorFormat #13734. Replaces the 2 s WaitAll with a 30 s drain-on-every-slice loop. Should cover both ToolTaskCanChangeCanonicalErrorFormat and ErrorWhenTextSentToStandardError.
• Shrink TaskNodesDieAfterBuild WaitForExit budget 15s -> 2s (follow-up to #13828) #13845 (draft) -- shrink TaskNodesDieAfterBuild budget 15 s -> 2 s. New data confirms the fix held (1 -> 0 over 6 days, p95 ~44 ms / max 191 ms). Recommending we flip this out of draft.

Persistent flakes still without a fix PR (top by post-window severity)

• MSBuildTaskHostTests.CompileNet35WinFormsApp (2 -> 3, multi-branch)
• FileStateTests.LastWriteTimeUtcOnDirectory (1 -> 3, multi-branch) -- was tracked in Flaky test: FileStateTests.LastWriteTimeUtcOnDirectory #13847, may need re-open
• BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput (2 -> 3) -- tracked in Flaky test BinaryLoggerShouldEmbedFilesViaTaskOutput #13762
• MSBuildServer_Tests.CanShutdownServerProcess (2 -> 2, multi-branch)
• MultithreadedExecution_Tests.MultithreadedBuild_Success / _BinaryLogging (2 -> 2 each)

Net-new flakes (mostly noise)

20 tests appeared post-merge that were not present pre-merge, but 14+ of them share the same single-branch / single-timestamp pattern (CodeTaskFactoryTests.* cluster all at 2026-05-25 06:45:13, several BuildCheck.UnitTests.EndToEndTests.* at the same minute) -- almost certainly one bad PR run, not real new flakiness. Ignoring those.

Genuinely worth a tracker: WriteLinesToFile_Tests.TransactionalModePreservesAllData (2 fails, 2 branches, new).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com