Skip to content

ParallelTests fails when one repetition runs after its siblings on a busy runner #7581

Closed

Description

@thomhurst

Summary

ParallelTests asserts, in an [After(Test)] hook, that every recorded test run overlaps some other run. On a busy runner the engine can legitimately schedule a repetition of a 5-second test after its siblings have finished, so that run overlaps nothing and the hook fails. Even with [Retry(3)], this failed the macOS AOT job on a PR that changes no engine code.

Reproduction

AOT (macos-latest) on #7576, run 41797132503, job 126463754193. ExpectedStateTests.Pass runs every [EngineTest(ExpectedResult.Pass)] test in one AOT process, so ParallelTests competes with the whole suite for parallel slots on the 3-core runner.

failed Parallel_Test3 (5s 063ms)   (5 times, including retries)
  TestExecutionException: After(Test) hook failed: Expected to be true but found False
  at Assert.That(TestDateTimeRanges.Except([testDateTimeRange]).Any(x => x.Overlap(testDateTimeRange))).IsTrue()

Source: tests/TUnit.TestProject/ParallelTests.cs, AssertOverlaps. #4876 (closed) described the same class of timing fragility.

Expected

The test proves that tests without [NotInParallel] can run concurrently, without requiring that every individual run overlaps another. For example, an [After(Class)] check that at least two runs overlapped at some point would hold however the scheduler orders the later repetitions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium: edge case, mode divergence, flaky test, or workaround existsbugSomething isn't workingin-progressWork is in progress

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions