Skip to content

ci(e2e): don't run the n-1/n-2 SDK legs on the nightly channel - #973

Merged
juhovainio merged 1 commit into
mainfrom
worktree-nightly-skip-nminus-lanes
Nov 9, 2026
Merged

juhovainio merged 1 commit into
mainfrom
worktree-nightly-skip-nminus-lanes

Conversation

@juhovainio

Copy link
Copy Markdown
Collaborator

Problem

On a release-branch push the e2e-gpu-strix-ubuntu lane crosses its channel: [release, nightly] axis with the sdk_version: [current, n-1, n-2] matrix. The two nightly legs for n-1 and n-2 fail in the pre-warm, before any scenario runs:

Error: failed to resolve TheRock nightly wheel runtime from canonical source https://rocm.nightlies.amd.com/whl-multi-arch
    no mutually compatible TheRock rocm, torch, torchvision, and torchaudio versions were found for version 7.14.1

(7.14.0 for n-2.) Seen on the release/v0.3.0 push, run 42247485118. The pins come from rocm_releases.rs and name released versions, but the nightly stream only carries the latest (pre-)release builds, so it cannot satisfy them. The same pins pass on the release channel.

Change

  • e2e-selfhosted.yml: exclude channel: nightly × sdk_version: n-1 and n-2 from the e2e-gpu-strix-ubuntu matrix. A release-branch push now runs four legs on that lane instead of six. On ordinary PRs the matrix is the single element current, so the exclude matches nothing.
  • docs/ci-hardware-testing.md: the "all six combinations" claim now says four, with the reason.

The released versions stay covered by the release-channel legs. The consolidated-report check only lists the current artifacts, so it does not expect the dropped uploads.

Verification

  • cargo test -p xtask passes, including the workflow contract tests. The workflow YAML parses with the expected exclude list.
  • Not run: actionlint (not installed locally), and the exclude has not been exercised on a real run. The first release-branch push after this lands is the check; this does not reach release/v0.3.0 until it is cherry-picked or merged there.
  • No scenario or regression test: this is CI plumbing with no CLI-observable behavior.

The cancelled W7900/MI300X jobs on that run are a separate runner-availability problem and are not touched here.

On a release-branch push, the Strix Halo Ubuntu lane crossed its
channel axis with the sdk_version matrix, so the nightly channel was
asked to pin the n-1 (7.14.1) and n-2 (7.14.0) released versions. The
nightly aggregate stream only carries the latest (pre-)release builds,
so `rocm install sdk` failed in the pre-warm with "no mutually
compatible TheRock rocm, torch, torchvision, and torchaudio versions"
before any scenario ran.

Exclude those two combinations from the matrix. The released versions
are still exercised by the release channel legs. Ordinary PR runs use
a single-element `current` matrix, so the exclude is a no-op there.

Signed-off-by: juhovainio <juho.vainio@amd.com>
@juhovainio
juhovainio requested a review from a team as a code owner November 9, 2026 09:39

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. This was a static review of the diff; I did not run the tests or the workflow.

I checked that the exclude keys match the matrix of e2e-gpu-strix-ubuntu (channel, sdk_version), that it removes exactly nightly x n-1 and n-2 (four of six legs on a release push, a no-op when the matrix is ["current"]), and that the consolidated-report check only expects the current artifacts, so nothing waits on the dropped uploads. No blockers.

Non-blocking:

  • A contract test for the exclude would be nice, see the inline comment.
  • The Triggers paragraph in docs/ci-hardware-testing.md still opens with "every self-hosted GPU job additionally expands into ... three legs" before the strix-ubuntu caveat a few lines later; a short "(except nightly on strix-ubuntu, below)" would avoid a universal claim that is now false for one lane (AGENTS.md §5: grep for the old claim).
  • The nightly-stream explanation is inferred from the pre-warm error; "appears to" would be more accurate wording.

🤖 by agenthub on AMD AgentHub

# (`rocm install sdk` fails with "no mutually compatible TheRock ...
# versions"). Those legs are covered by the `release` channel; a no-op
# when the matrix is the single-element `current`.
exclude:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: nothing pins this exclude. The repo pins sibling workflow behaviour in xtask/src/workflow_contract.rs, and AGENTS.md §3 asks that a new test is shown to fail when the code it covers is broken. If someone drops this block, the red nightly n-1/n-2 legs come back quietly. Consider a small contract test asserting that the e2e-gpu-strix-ubuntu matrix excludes nightly x n-1 and n-2, or say in the PR text that you chose not to add one.

@juhovainio
juhovainio added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 9e78f6b Nov 9, 2026
37 checks passed
@juhovainio
juhovainio deleted the worktree-nightly-skip-nminus-lanes branch November 9, 2026 12:00
juhovainio added a commit that referenced this pull request Nov 9, 2026
On a release-branch push, the Strix Halo Ubuntu lane crossed its
channel axis with the sdk_version matrix, so the nightly channel was
asked to pin the n-1 (7.14.1) and n-2 (7.14.0) released versions. The
nightly aggregate stream only carries the latest (pre-)release builds,
so `rocm install sdk` failed in the pre-warm with "no mutually
compatible TheRock rocm, torch, torchvision, and torchaudio versions"
before any scenario ran.

Exclude those two combinations from the matrix. The released versions
are still exercised by the release channel legs. Ordinary PR runs use
a single-element `current` matrix, so the exclude is a no-op there.

Signed-off-by: juhovainio <juho.vainio@amd.com>
(cherry picked from commit 9e78f6b)
Signed-off-by: juhovainio <juho.vainio@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants