Skip to content

ci: move the install lifecycle and self-test steps out of build-and-test so E2E starts sooner #871

Open

Description

@jussielo-amd

Is your feature request related to a problem? Please describe.

After #810 split the Windows build into two parallel jobs, the Windows lane is no longer what sets the length of a CI run. Over the 20 successful push runs on main since that merge, the Windows lane finished at a median of 8.3 minutes after the run started, while the whole run took a median of 13.3 minutes. The remaining critical path is changes -> clippy -> build-and-test -> E2E tests -> E2E consolidated report. In those runs build-and-test started at about 2.9 minutes and ended at about 6.5, so E2E tests only started at about 6.9 minutes.

E2E tests uses build-and-test only as a gate (it checks the job's result and does not consume any of its output), but build-and-test also runs several steps that nothing downstream waits for. Median step durations over the same 20 runs: Acceptance install lifecycle 106s (range 76 to 161s), GPU reclaim self-test 31s and Pinned key consistency check 10s, out of a median job time of 218s. Together that is about 147s of a job that otherwise takes about 70s (checkout, native deps, toolchain, cargo build --workspace at 26s on a warm sccache, and the local smoke test).

Describe the solution you'd like

Move the steps that E2E tests does not depend on out of build-and-test into a sibling job on the same trigger conditions: Acceptance install lifecycle, GPU reclaim self-test, Pinned key consistency check, and the three near-instant self-test steps (Acceptance harness self-tests, Portable WSL build deps self-test, Release readiness self-test). build-and-test keeps the build and the local no-fallback smoke test, so E2E tests can start about 2.4 minutes earlier. This is the same shape as the windows-debug / windows-release split. The expected saving is an estimate from step medians, not something I have run on a branch.

Open points that need an answer before or while implementing this:

  • Required checks. If build-and-test is a required status check, the new job has to become one too, otherwise a failing lifecycle run would stop blocking merges. That needs someone with admin rights on the repository. Keeping build-and-test as an aggregator of the two jobs, as windows-build-and-test does, does not help here, because E2E tests would then wait for the slow job again.
  • E2E gate semantics. E2E tests would no longer be skipped when the lifecycle steps fail. That is fine as long as the new job is itself required.
  • Build dependence. cargo xtask e2e builds its own release binaries (cargo build --release -p rocm -p rocmd), so the lifecycle step does not reuse the debug build from build-and-test. What the sibling job gives up is the warm dev-profile dependencies for the e2e harness, the same trade-off documented for windows-release. Cold-cache duration (fork pull requests, first run after the split) has not been measured.
  • Cache pressure. The rust-cache key includes the job id, so a new job starts cold and orphans the old entry, and the workflow comments note the Actions cache is already at its size cap.
  • Contract tests and docs. xtask/src/workflow_contract.rs pins the Windows lane in detail; a new Linux gate with a required result would need equivalent pins, and the comments and docs that describe what build-and-test runs would need updating.

Describe alternatives you've considered

  • Do nothing beyond the companion change that stops build-and-test from waiting for clippy. That change alone is estimated to bring the median run from about 13.3 to about 11 minutes and has none of the open points above.
  • Change the needs of E2E tests so it runs alongside build-and-test. This is the largest saving on paper, but it changes the E2E gate and the gain is small once the Windows lane (8.3 minutes) becomes the floor.

Additional context

Suggested sequencing: land the clippy wait change first, re-measure over at least 20 push runs on main, and only then decide whether this split is still worth its cost. After both changes the Windows lane and Test (affected crates) would be close to the critical path, so a further saving would need work on those as well. Run timings above come from the CI workflow's job and step durations via the GitHub API for push runs on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions