Skip to content

CI: PublishTestProjectAOTModule intermittently hangs for 30 minutes on macOS #7211

Closed

Description

@thomhurst

What happened

On the modularpipeline (macos-latest) job, PublishTestProjectAOTModule sometimes runs until the 30-minute module timeout and is then killed:

PublishTestProjectAOTModule ✗ (30.0m) - ModuleTimeoutException
ModuleTimeoutException: PublishTestProjectAOTModule has timed out after 30m & 0s. The module did not respond to the cancellation token and was forcibly ...

The same module normally finishes in about 1.8 minutes on macOS, so this is a hang, not a slow publish. Seen on 2026-10-20 on three unrelated PRs:

Rerunning the failed job passes.

Where

tools/TUnit.Pipeline/Modules/PublishTestProjectAOTModule.cs, which runs dotnet publish with Aot=true for TUnit.TestProject (net10.0) into artifacts/aot-publish.

Expected

The publish either completes or fails fast. Worth checking:

  • whether the native linker / ILCompiler step stalls under the parallel load of other modules on the macOS runner (memory pressure);
  • adding ShowStandardOutput or a heartbeat so the hang point is visible in logs;
  • whether dotnet publish should get --disable-build-servers or a lower ILC parallelism on macOS;
  • whether the module should honour cancellation so the job fails at the timeout instead of being force-killed.

Reproduce

Intermittent on GitHub-hosted macos-latest runners during a busy queue.

Activity

  1. thomhurst commented on Oct 20, 2026

    @thomhurst
    OwnerAuthor

    Seen again on PR #7180 macOS (job 118476810599).

  2. thomhurst commented on Oct 20, 2026

    @thomhurst
    OwnerAuthor

    Seen again on PR #7144 macOS (job 118485945080). Fifth occurrence on 2026-10-20.

  3. added
    P2Medium: edge case, mode divergence, flaky test, or workaround exists
    on Oct 31, 2026
  4. thomhurst commented on Nov 1, 2026

    @thomhurst
    OwnerAuthor

    Closing: the cause is fixed on main. These were not hangs. After #7131 the AOT publish competed with the test modules on the 3-core macOS runner and took 24–27 minutes even on passing runs. #7216 raised the module timeout to 60 minutes, and #7226 moved the AOT publish into its own aot.yml workflow, so it no longer competes with the test modules.

    Evidence: of the last 60 aot.yml runs, 46 succeeded, 5 were cancelled and 2 failed. Neither failure was a timeout: every failed job finished in 4–8 minutes (runs 41307113428 and 41201223625). Reopen this if a ModuleTimeoutException shows up again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium: edge case, mode divergence, flaky test, or workaround existsbugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions