Repository navigation
ci(e2e): don't run the n-1/n-2 SDK legs on the nightly channel - #973
Conversation
On a release-branch push, the Strix Halo Ubuntu lane crossed its channel axis with the sdk_version matrix, so the nightly channel was asked to pin the n-1 (7.14.1) and n-2 (7.14.0) released versions. The nightly aggregate stream only carries the latest (pre-)release builds, so `rocm install sdk` failed in the pre-warm with "no mutually compatible TheRock rocm, torch, torchvision, and torchaudio versions" before any scenario ran. Exclude those two combinations from the matrix. The released versions are still exercised by the release channel legs. Ordinary PR runs use a single-element `current` matrix, so the exclude is a no-op there. Signed-off-by: juhovainio <juho.vainio@amd.com>
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good. This was a static review of the diff; I did not run the tests or the workflow.
I checked that the exclude keys match the matrix of e2e-gpu-strix-ubuntu (channel, sdk_version), that it removes exactly nightly x n-1 and n-2 (four of six legs on a release push, a no-op when the matrix is ["current"]), and that the consolidated-report check only expects the current artifacts, so nothing waits on the dropped uploads. No blockers.
Non-blocking:
- A contract test for the
excludewould be nice, see the inline comment. - The Triggers paragraph in
docs/ci-hardware-testing.mdstill opens with "every self-hosted GPU job additionally expands into ... three legs" before the strix-ubuntu caveat a few lines later; a short "(except nightly on strix-ubuntu, below)" would avoid a universal claim that is now false for one lane (AGENTS.md §5: grep for the old claim). - The nightly-stream explanation is inferred from the pre-warm error; "appears to" would be more accurate wording.
🤖 by agenthub on AMD AgentHub
| # (`rocm install sdk` fails with "no mutually compatible TheRock ... | ||
| # versions"). Those legs are covered by the `release` channel; a no-op | ||
| # when the matrix is the single-element `current`. | ||
| exclude: |
There was a problem hiding this comment.
Non-blocking: nothing pins this exclude. The repo pins sibling workflow behaviour in xtask/src/workflow_contract.rs, and AGENTS.md §3 asks that a new test is shown to fail when the code it covers is broken. If someone drops this block, the red nightly n-1/n-2 legs come back quietly. Consider a small contract test asserting that the e2e-gpu-strix-ubuntu matrix excludes nightly x n-1 and n-2, or say in the PR text that you chose not to add one.
On a release-branch push, the Strix Halo Ubuntu lane crossed its channel axis with the sdk_version matrix, so the nightly channel was asked to pin the n-1 (7.14.1) and n-2 (7.14.0) released versions. The nightly aggregate stream only carries the latest (pre-)release builds, so `rocm install sdk` failed in the pre-warm with "no mutually compatible TheRock rocm, torch, torchvision, and torchaudio versions" before any scenario ran. Exclude those two combinations from the matrix. The released versions are still exercised by the release channel legs. Ordinary PR runs use a single-element `current` matrix, so the exclude is a no-op there. Signed-off-by: juhovainio <juho.vainio@amd.com> (cherry picked from commit 9e78f6b) Signed-off-by: juhovainio <juho.vainio@amd.com>
Problem
On a release-branch push the
e2e-gpu-strix-ubuntulane crosses itschannel: [release, nightly]axis with thesdk_version: [current, n-1, n-2]matrix. The two nightly legs for n-1 and n-2 fail in the pre-warm, before any scenario runs:(7.14.0 for n-2.) Seen on the
release/v0.3.0push, run 42247485118. The pins come from rocm_releases.rs and name released versions, but the nightly stream only carries the latest (pre-)release builds, so it cannot satisfy them. The same pins pass on thereleasechannel.Change
e2e-selfhosted.yml: excludechannel: nightly×sdk_version: n-1andn-2from thee2e-gpu-strix-ubuntumatrix. A release-branch push now runs four legs on that lane instead of six. On ordinary PRs the matrix is the single elementcurrent, so the exclude matches nothing.docs/ci-hardware-testing.md: the "all six combinations" claim now says four, with the reason.The released versions stay covered by the release-channel legs. The consolidated-report check only lists the
currentartifacts, so it does not expect the dropped uploads.Verification
cargo test -p xtaskpasses, including the workflow contract tests. The workflow YAML parses with the expectedexcludelist.actionlint(not installed locally), and theexcludehas not been exercised on a real run. The first release-branch push after this lands is the check; this does not reachrelease/v0.3.0until it is cherry-picked or merged there.The cancelled W7900/MI300X jobs on that run are a separate runner-availability problem and are not touched here.