Repository navigation
ci(e2e): add a W7900 self-hosted E2E lane (EAI-8763) - #454
Conversation
jussielo-amd
left a comment
There was a problem hiding this comment.
The nightly lane's GPU-preflight comment contradicts the per-PR lane's comment added in this same PR.
nightly.yml:693describes the 16 GiB floor as "a third of this card's 48GB" — a proportional framing.e2e-selfhosted.yml:1367-1371(added in this same diff) explicitly states the floor is "a flat constant rather than a proportion of capacity" and warns not to scale it down for a smaller card.
Required fix: update the nightly.yml:693 comment to match the flat-constant rationale in e2e-selfhosted.yml, so a future reader doesn't "fix" the floor into scaling with VRAM capacity based on the wrong comment.
Rest of the PR looks correct — workflow structure, needs/if gating, e2e-report label mapping, and docs all check out against the existing rad3/mi350p pattern.
|
🔴 Automated review · pr-review-watcher · 8d4f0a0 This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryAdds a self-hosted W7900 (gfx1100) end-to-end lane plus its nightly twin, wired into the consolidated report, with a new slug pin and doc/contract updates. Outcome: Needs work — this head does not discharge either change request currently open against it: the contradictory nightly preflight comment is untouched, and both new upload steps still sit at the default 🚫 Blocking (must fix before merge)
Non-blocking
|
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 48bf12c
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Two things to fix before this lane is trustworthy.
The lane produces no failure signal for a whole class of failures. .github/workflows/e2e-selfhosted.yml:1345 and the nightly twin — with continue-on-error: true, a failure is tolerated in the Checks UI; and because the upload steps leave if-no-files-found at its default of warn, a run that dies before report.json exists (preflight failure, build failure, a crash before results are written) uploads nothing, and the consolidated report omits a platform with no report.json rather than flagging it. The lane then shows green and vanishes from the report — no row, no FAIL, nothing for a human to notice. This repo already wrote that hazard down in nightly.yml and guarded against it with if-no-files-found: error on the WSL copy-out step, but the guard was never extended to a GPU lane's own upload step; for the five lanes at continue-on-error: false the red check still covers it, and for this new lane nothing does. Adding if-no-files-found: error to both new upload steps closes it. Worth applying to the MI350P lane in the same pass — it has the identical exposure. For the record, assertion failures that do reach report.json are rendered truthfully as FAIL and are not masked; the hole is specifically pre-report failures.
The nightly twin's preflight comment contradicts the per-PR lane's, in the same commit. .github/workflows/nightly.yml:693-695 still reads "the shared 16 GiB default, a third of this card's 48GB … Same as the per-PR W7900 lane", while the per-PR comment added by this same commit says the floor "is a flat constant rather than a proportion of capacity … do NOT scale it down for a smaller part." Only one of the two instances was corrected, and the uncorrected one points at the corrected one as its authority. This matters concretely: the stated reason the runner label is per-SKU is that a same-family 24GB card must not inherit this lane's sizing, and the proportional framing is exactly what would invite someone adding that lane to scale the floor down.
The full findings, including four non-blocking items and a note on why a two-dot diff misreads this branch, are in the review comment on this PR.
48bf12c to
8d4f0a0
Compare
Superseded — re-filing with the one item that still stands, plus a correction to a remedy in this request that does not actually work. See the new request.
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 8d4f0a0
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
This supersedes my earlier change request on this pull request, which has been withdrawn. One item still stands, and one item in the old request carried a remedy of mine that does not work; both are set out below.
Still blocking
.github/workflows/nightly.yml:693-695 — the nightly twin's preflight comment contradicts the per-PR lane's, in the same commit. It reads "Floor is the shared 16 GiB default, a third of this card's 48GB … Same as the per-PR W7900 lane", while the per-PR comment added by this same commit (e2e-selfhosted.yml:1657-1663) says the floor "is a flat constant rather than a proportion of capacity … do NOT scale it down for a smaller part."
This is the twin shape this file keeps producing — a correction applied to one of two near-identical lanes — and here the uncorrected twin cites the corrected one as its authority. It matters concretely: the stated reason the runner label is per-SKU is that a same-family 24GB card must not inherit this lane's sizing, and the proportional framing is exactly what would invite whoever adds that lane to scale the floor down. The wording to copy already exists in-tree: the mi350p lane says "matches the other Instinct lane rather than scaling with the 144GB capacity".
Fix: reword nightly.yml:693-695 to the flat-constant rationale used by its own per-PR sibling.
Correcting my own earlier remedy
The previous request called the default if-no-files-found on the new upload steps blocking and prescribed if-no-files-found: error. That prescription does not close the gap: with job-level continue-on-error: true, a failed upload step is tolerated too and only produces a log annotation, so the check stays green either way. What actually closes it is flipping continue-on-error to false once the hardware has a track record — which this PR's own docs change already time-boxes — or having the consolidated report flag an expected-but-absent platform, for which the expected set is already enumerated in CANONICAL_REPORT_ARTIFACTS. Since the exposure is the same one the shipped mi350p lane already carries and is explicitly time-boxed here, it is a known accepted gap rather than a new defect, and it is no longer blocking. Apologies for the round-trip.
The remaining observations are in the review comment and none of them block.
|
Rebased onto Blocking (@jussielo-amd, @siloteemu): the nightly W7900 preflight comment now uses the per-PR lane's flat-constant rationale. Both rad3 comments ("half of the R9700S's 32GB") had the same proportional framing and are corrected too. Non-blocking, all taken:
I've also corrected the description: it wrongly said Strix WSL2 failed on |
|
Nice work here — this is a thorough, well-substantiated PR. The technical claims in the description (the Lemonade-vs-vLLM engine selection, the Findings Blocking: none. Verified against source (not just the diff):
Non-blocking / already-acknowledged tradeoffs:
Positive signals:
|
|
🔴 Automated review · pr-review-watcher · 2c8c5a8 This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer. SummaryAdds a self-hosted gfx1100 workstation E2E lane and its nightly twin, wires both into the consolidated report, and makes both consolidated-report jobs fail when a lane that ran uploaded no report artifact. Outcome: No blocking findings — the remediation discharges the change request in full and the new guard is genuinely tested rather than merely asserted. Verified: on a scratch copy 🚫 Blocking (must fix before merge)None. Non-blocking
|
siloteemu
left a comment
There was a problem hiding this comment.
Review request seen. The last automated review covered this exact head (2c8c5a8, no blocking findings, 3 non-blocking): #454 (comment)
Nothing reviewable has changed since (same head, same description), so no new review was run; a push will trigger one.
The AMD Radeon PRO W7900 (gfx1100, 48GB) had no E2E coverage at any cadence. This adds the per-PR lane and its nightly twin, wired into the consolidated report. The runner label is `w7900` rather than a `gfx1100` family label. The RX 7900 XTX shares that ISA but carries half the VRAM, and GitHub dispatches a job to any runner whose labels are a superset of the job's, so a family label would send 48GB-sized work to a 24GB card -- the mismatch the per-hardware label convention exists to prevent. The preflight floor stays at the shared 16 GiB default. That value is a flat constant, not a proportion of VRAM: e2e-gpu and e2e-gpu-mi350p use the same 16 GiB against very different capacities, because the floor only has to catch a leftover serve while letting a clean host pass on the first poll. Both lanes are non-blocking while the hardware is proven out, matching how the MI350P lane was introduced. `gfx1100` slugs as itself through `normalize_family` and `platform_hardware_slug`, exactly as `gfx1201` does for the R9700 lane, so the report needs only the artifact label mapping. That fallthrough is the absence of a rule, so `platform_slug_derivation` now pins both families: `rocm_core::normalize_therock_family` folds them into `gfx110X-all`/`gfx120X-all` for wheel selection, and aligning the two functions would route both lanes' artifacts to `e2e-unknown-report` with every test still green. Note the coverage this adds is Lemonade-shaped: `family_prefers_vllm` gates on data-center families, so `gfx1100` resolves to lemonade and the `@requires-engine:vllm` scenarios skip -- the same profile the shipped R9700 lane has. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The nightly W7900 preflight described the 16 GiB floor as a third of the card's 48GB, while its per-PR twin says the floor is a flat constant that must not scale with capacity. The proportional wording is exactly what would invite whoever adds a same-family 24GB lane to lower the floor, so it now carries the per-PR rationale. Both rad3 preflight comments framed the same floor as half of the card's 32GB and get the same correction. The per-PR reclaim comment now says what the MI350P twin says: the runner scales to zero between jobs but its work volume persists. The contract test's message tied "not blocking" to continue-on-error, contradicting the workflow header; it now names the real reason, that no self-hosted lane is a required status check. The large-model docs named only MI300X and Strix Halo. The model actually follows the lane's serve engine, so MI350P gets the vLLM model and R9700 and W7900 get the Lemonade checkpoint; the README, the feature comment and the step comment now say so. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A lane that dies before writing report.json (preflight, build, a crash before results exist) uploads nothing, and the consolidated grid can only render reports that arrived, so the lane dropped out with no row and no FAIL. On a continue-on-error lane -- the W7900 and MI350P per-PR lanes and every nightly lane -- nothing else turned red either, so the failure was invisible. Both report jobs now check that every lane that ran (any `needs` result but skipped) uploaded its report artifact, reading the run's artifact list rather than the download directory, which flattens a single match and loses its name. A missing report fails the job with the lane named in an annotation and the step summary; an API failure says so rather than dying without a word. The check can miss a re-run that uploads nothing, because earlier attempts' artifacts stay listed, but it never flags a lane that did upload. The contract tests pin the lane-to-artifact map against each lane's upload step, the step's guard and inputs, and the per-PR and nightly copies of the script against each other. A behaviour test runs the script under `bash -e` with a stub `gh` across every lane reported, a lane that ran and uploaded nothing (including as success or cancelled), a skipped lane, lanes after a skipped one, two lanes missing at once, a lane absent from `needs`, a timed-out lane that still uploaded, a near-miss name and an API failure; on CI a missing `jq` fails it rather than skipping. The WSL upload guard's comment is rewritten to match, since its claim that the report never flags an absent platform no longer holds. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
2c8c5a8 to
07021f8
Compare
Main moved under this branch: per-PR rad3/mi350p lanes were removed, the GPU reclaim became scripts/reclaim-gpu.sh, preflights poll at least once before their deadline and report holders, and the nightly lanes and the strix-ubuntu lane gained a release/nightly channel matrix. Apply the same to both W7900 lanes, and teach the missing-report check to see a channel-matrix lane through its release leg, since the report job has no matrix context. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
07021f8 to
18abd87
Compare
Rebase picks up #454's W7900 lane (PR #415/#454/#463 all now merged). Drop the hardcoded '4 jobs'/'4 platforms' counts from the validation ladder table so they can't drift again as lanes are added, and credit ci_yml_schedules_no_self_hosted_job alongside self_hosted_workflow_owns_the_gpu_lanes for what each test actually checks. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…Cm#465) * docs(ci): document the three-stage validation ladder Add the missing framing to docs/ci-hardware-testing.md: the three self-hosted GPU gates (per-PR smoke, nightly coverage, release-candidate regression) and how they differ; the validation axes (OS, hardware, channel, SDK-version); why the serve engine isn't an independently selectable matrix axis (quotes effective_serve_engine()); and a short addendum on e2e-prewarm's --version/--build-date pin flags. No lane, artifact, or job-order changes, so every contract-tested table/list/ sentence in the doc is left byte-identical. ROCMAI-431 Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci): fix e2e-prewarm tense, reword two overstated doc claims Review found the --version/--build-date paragraph claiming present-tense availability for a flag pair that isn't in this tree yet (ships in ROCm#464, stacked on ROCm#415, neither merged) — contradicted the ladder section's own "ships in ROCm#464" framing two paragraphs up. - e2e-prewarm --version/--build-date: reworded to future tense, matching the ladder section. - Per-PR row: note e2e-gpu-strix-ubuntu already runs both channel legs per PR, not just 4 flat jobs. - "cannot be silently undone": reworded to match what self_hosted_workflow_owns_the_gpu_lanes actually checks (job keys present as text, not a structural pin). Left the two open-decisions-for-the-author items untouched per review (non-defects). Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci): reword ROCm#415/ROCm#464 merge-status passages now that ROCm#415 landed PR ROCm#415's release/** trigger merged 2026-10-01; e2e-selfhosted.yml on main already triggers on push: branches: [main, "release/**"]. Only ROCm#464 (the SDK pin) is still pending. Fixes the two passages in the ladder section and the pre-warm section that still said neither PR was merged. Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci): fix engine-selection doc errors, de-stale the ROCm#464 status prose pr-review-watcher flagged 3 blocking issues at f679385: - the engine section said rocm serve picks its engine via the harness's effective_serve_engine(), when capability.rs's own header says that function is the re-implementation and select_serve_engine (main.rs) is the product original. - "the engine is not selectable independent of hardware and OS" overstated it: select_serve_engine checks an explicit --engine and a configured default_engine before falling back to the GPU-family preference, and a scenario can pin either side with @requires-engine:<vllm|lemonade>. - the ROCm#464 merge-status prose (two copies) asserted "still pending as of this writing", which goes false the moment ROCm#464 merges. Reworded both to state what the tree supports today instead of PR status. Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci): rebase onto main, fix stale citations and self-contradiction - rebase onto main (PR ROCm#415/ROCm#460/ROCm#463 already merged there) - cite select_serve_engine's real location (serve_cmd.rs, not main.rs) and turn the three new bare-backtick file citations into relative markdown links per AGENTS.md's citation rule - drop the sentence implying e2e-prewarm already has --version/--build-date flags; state once (in the ladder section) that the pin is pending work (ROCMAI-430) and cross-reference it from the pre-warm addendum instead of repeating it - describe select_serve_engine's full precedence (recipe-support gate, recipe's own preferred engine, platform default), not just the two-step simplification - say 'native Windows' for the lemonade-only engine case; WSL reports os_family linux and stays vLLM-eligible Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci-hardware-testing): link the therock.rs citation per AGENTS.md §5 A bare backtick path doesn't get checked by docs-links, so it can go stale silently. Link it relative to this file, and name the actual private function instead of just the file. Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci): rebase onto main and fix stale lane counts and test credit Rebase picks up ROCm#454's W7900 lane (PR ROCm#415/ROCm#454/ROCm#463 all now merged). Drop the hardcoded '4 jobs'/'4 platforms' counts from the validation ladder table so they can't drift again as lanes are added, and credit ci_yml_schedules_no_self_hosted_job alongside self_hosted_workflow_owns_the_gpu_lanes for what each test actually checks. Signed-off-by: Juho Vainio <juho.vainio@amd.com> * docs(ci): trim the duplicated e2e-prewarm pin-status explanation The pre-warm section restated the no-pin/ROCMAI-430 fact that the validation-ladder section already explains, contradicting the PR's own claim that it's stated once. Cross-reference instead of repeating it. Signed-off-by: Juho Vainio <juho.vainio@amd.com> --------- Signed-off-by: Juho Vainio <juho.vainio@amd.com>
The AMD Radeon PRO W7900 (gfx1100, 48GB) had no E2E coverage at any cadence. This adds the per-PR lane and its nightly twin, wired into the consolidated report. Following review, it also makes the consolidated report fail when a lane that ran uploaded no report (last section below).
Modelled on the MI350P lane.
The label is
w7900, notgfx1100The RX 7900 XTX is the same gfx1100 ISA with half the VRAM (24GB vs 48GB). GitHub dispatches a job to any runner whose labels are a superset of the job's, so a family label would send 48GB-sized work to a 24GB card — the mismatch the per-hardware label convention exists to prevent. An XTX pool would get its own label.
What the coverage actually is
Worth stating plainly rather than leaving to be discovered: this adds Lemonade coverage, not vLLM coverage.
family_prefers_vllmgates on data-center families, sogfx1100resolves toeffective_serve_engine == "lemonade"and the@requires-engine:vllmscenarios skip rather than run. The@nightlylarge-model scenario takes the Lemonade branch — the Q4 35B-A3B GGUF, comfortably inside 48GB.This is the same profile the shipped R9700 lane has, so it is precedented. I specifically checked whether the nightly lane would attempt the ~54 GiB bf16 model on a 48GB card and it will not: engine selection is a positive allowlist of data-center parts that
gfx1100never reaches.One test pin that is not about this lane alone
gfx1100slugs as itself throughnormalize_familyandplatform_hardware_slug, exactly asgfx1201does for R9700, so the report needs only the artifact mapping — nocapability.rslogic change.But that fallthrough is the absence of a rule, and nothing pinned it.
rocm_core::normalize_therock_family— similarly named, and documented as being mirrored "in subset" by the harness one — does foldgfx1100/gfx1201intogfx110X-all/gfx120X-allfor wheel selection. A future maintainer aligning the two would send both this lane's and rad3's artifacts toe2e-unknown-reportwith every test in the repo still green.platform_slug_derivationnow pins both families. Verified by mutation: adding agfx11*arm tonormalize_familymakes it fail withgfx targetgfx1100must slug asgfx1100``. This closes a pre-existing gap that rad3 shipped with.Preflight floor
Stays at the shared 16 GiB default. That is a flat constant, not a proportion of VRAM —
e2e-gpuande2e-gpu-mi350puse the same 16 GiB against very different capacities, and the MI350P lane's own comment says so explicitly. The floor only has to catch a leftover serve while letting a clean host pass on the first poll. An earlier revision of this branch justified it as "proportional headroom", which was simply wrong, and is corrected in the comment. The nightly twin and both rad3 comments carried the same proportional framing and now say the same thing.While there, the workflow header's claim that
continue-on-errorisfalsefor every lane is now accurate: it isfalseon five of seven, withe2e-gpu-mi350pand this lanetrueonly while their hardware is proven out.Validation
cargo test -p xtask157 passed,-p e2e-cucumber --lib125 passed,-p e2e-report --lib46 passed. The contract guards are derived from the YAML, so they cover this lane automatically — including the docs guard, whose prose anchors are matched as literal contiguous strings.prek run --all-filesclean.scripts/reclaim-gpu.shdoes not exist on this base — if ci(e2e): reclaim engine processes leaked into the pre-warm tree (EAI-8751) #406 lands first, this lane should be rebased onto the script with the others.Result on hardware
The lane passed on its first run —
E2E tests (W7900), conclusionsuccess, 11:37:03Z to 11:57:21Z (~20 min) on runnerw7900-0.I flagged two runtime preconditions that could not be checked from source, and noted
continue-on-error: truewould mask either if it were wrong. Both are now answered by that run, since either would have failed it:releasechannel does ship usablegfx110X-allartifacts —cargo xtask e2e-prewarm --channel releasecompleted.The runner is scale-to-zero, so this also exercised the full path rather than a warm host: it was at zero replicas when the job queued, woke on the queue, ran, and released the GPU afterwards.
Strix Halo lanes
Rebased onto
main, which picked up #442's GTT-aware Strix preflight. Both Strix lanes have since passed on this PR. Strix Ubuntu first needed one re-run: its first attempt was cancelled at the 45-minute cap after Lemonade spent ~17 minutes retrying two upstream downloads that were returning 404 and 403. An earlier version of this section said Strix WSL2 failed identically onmain; that was wrong, since WSL2 passed onmainat the old base.A lane that dies before reporting is no longer invisible
With
continue-on-error: true(this lane, MI350P, and every nightly lane), a run that dies before writingreport.jsonuploads nothing, and the consolidated grid can only render the reports that arrived. The lane showed green and dropped out of the grid.Both consolidated report jobs now end with a step that lists the run's artifacts and fails, naming the lane in an annotation and the step summary, when a lane that ran (any
needsresult butskipped) uploaded no report. The report jobs gainactions: readfor that. Known limits: a re-run that uploads nothing can pass on an earlier attempt's artifact, and the check tests for the artifact, not for areport.jsoninside it.Contract tests pin the lane-to-artifact map to each lane's upload step, the step's guard and inputs, and the per-PR and nightly copies of the script to each other. A behaviour test runs the script under
bash -ewith a stubbedgh. Each branch of the script was broken in turn to confirm the test fails.