Skip to content

ci(e2e): add a W7900 self-hosted E2E lane (EAI-8763) - #454

Merged
tomastola merged 4 commits into
mainfrom
ci/e2e-w7900-lane
Oct 15, 2026
Merged

tomastola merged 4 commits into
mainfrom
ci/e2e-w7900-lane

Conversation

@tomastola

@tomastola tomastola commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

The AMD Radeon PRO W7900 (gfx1100, 48GB) had no E2E coverage at any cadence. This adds the per-PR lane and its nightly twin, wired into the consolidated report. Following review, it also makes the consolidated report fail when a lane that ran uploaded no report (last section below).

Modelled on the MI350P lane.

The label is w7900, not gfx1100

The RX 7900 XTX is the same gfx1100 ISA with half the VRAM (24GB vs 48GB). GitHub dispatches a job to any runner whose labels are a superset of the job's, so a family label would send 48GB-sized work to a 24GB card — the mismatch the per-hardware label convention exists to prevent. An XTX pool would get its own label.

What the coverage actually is

Worth stating plainly rather than leaving to be discovered: this adds Lemonade coverage, not vLLM coverage. family_prefers_vllm gates on data-center families, so gfx1100 resolves to effective_serve_engine == "lemonade" and the @requires-engine:vllm scenarios skip rather than run. The @nightly large-model scenario takes the Lemonade branch — the Q4 35B-A3B GGUF, comfortably inside 48GB.

This is the same profile the shipped R9700 lane has, so it is precedented. I specifically checked whether the nightly lane would attempt the ~54 GiB bf16 model on a 48GB card and it will not: engine selection is a positive allowlist of data-center parts that gfx1100 never reaches.

One test pin that is not about this lane alone

gfx1100 slugs as itself through normalize_family and platform_hardware_slug, exactly as gfx1201 does for R9700, so the report needs only the artifact mapping — no capability.rs logic change.

But that fallthrough is the absence of a rule, and nothing pinned it. rocm_core::normalize_therock_family — similarly named, and documented as being mirrored "in subset" by the harness one — does fold gfx1100/gfx1201 into gfx110X-all/gfx120X-all for wheel selection. A future maintainer aligning the two would send both this lane's and rad3's artifacts to e2e-unknown-report with every test in the repo still green.

platform_slug_derivation now pins both families. Verified by mutation: adding a gfx11* arm to normalize_family makes it fail with gfx target gfx1100must slug asgfx1100``. This closes a pre-existing gap that rad3 shipped with.

Preflight floor

Stays at the shared 16 GiB default. That is a flat constant, not a proportion of VRAM — e2e-gpu and e2e-gpu-mi350p use the same 16 GiB against very different capacities, and the MI350P lane's own comment says so explicitly. The floor only has to catch a leftover serve while letting a clean host pass on the first poll. An earlier revision of this branch justified it as "proportional headroom", which was simply wrong, and is corrected in the comment. The nightly twin and both rad3 comments carried the same proportional framing and now say the same thing.

While there, the workflow header's claim that continue-on-error is false for every lane is now accurate: it is false on five of seven, with e2e-gpu-mi350p and this lane true only while their hardware is proven out.

Validation

  • cargo test -p xtask 157 passed, -p e2e-cucumber --lib 125 passed, -p e2e-report --lib 46 passed. The contract guards are derived from the YAML, so they cover this lane automatically — including the docs guard, whose prose anchors are matched as literal contiguous strings.
  • prek run --all-files clean.
  • The reclaim block is byte-identical to its siblings. scripts/reclaim-gpu.sh does not exist on this base — if ci(e2e): reclaim engine processes leaked into the pre-warm tree (EAI-8751) #406 lands first, this lane should be rebased onto the script with the others.

Result on hardware

The lane passed on its first run — E2E tests (W7900), conclusion success, 11:37:03Z to 11:57:21Z (~20 min) on runner w7900-0.

I flagged two runtime preconditions that could not be checked from source, and noted continue-on-error: true would mask either if it were wrong. Both are now answered by that run, since either would have failed it:

  • TheRock's release channel does ship usable gfx110X-all artifacts — cargo xtask e2e-prewarm --channel release completed.
  • The Lemonade backend for that family does run on discrete RDNA3, not only the Strix Halo APU path it was previously proven on.

The runner is scale-to-zero, so this also exercised the full path rather than a warm host: it was at zero replicas when the job queued, woke on the queue, ran, and released the GPU afterwards.

Strix Halo lanes

Rebased onto main, which picked up #442's GTT-aware Strix preflight. Both Strix lanes have since passed on this PR. Strix Ubuntu first needed one re-run: its first attempt was cancelled at the 45-minute cap after Lemonade spent ~17 minutes retrying two upstream downloads that were returning 404 and 403. An earlier version of this section said Strix WSL2 failed identically on main; that was wrong, since WSL2 passed on main at the old base.

A lane that dies before reporting is no longer invisible

With continue-on-error: true (this lane, MI350P, and every nightly lane), a run that dies before writing report.json uploads nothing, and the consolidated grid can only render the reports that arrived. The lane showed green and dropped out of the grid.

Both consolidated report jobs now end with a step that lists the run's artifacts and fails, naming the lane in an annotation and the step summary, when a lane that ran (any needs result but skipped) uploaded no report. The report jobs gain actions: read for that. Known limits: a re-run that uploads nothing can pass on an earlier attempt's artifact, and the check tests for the artifact, not for a report.json inside it.

Contract tests pin the lane-to-artifact map to each lane's upload step, the step's guard and inputs, and the per-PR and nightly copies of the script to each other. A behaviour test runs the script under bash -e with a stubbed gh. Each branch of the script was broken in turn to confirm the test fails.

@tomastola
tomastola requested a review from a team as a code owner September 28, 2026 11:32

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nightly lane's GPU-preflight comment contradicts the per-PR lane's comment added in this same PR.

  • nightly.yml:693 describes the 16 GiB floor as "a third of this card's 48GB" — a proportional framing.
  • e2e-selfhosted.yml:1367-1371 (added in this same diff) explicitly states the floor is "a flat constant rather than a proportion of capacity" and warns not to scale it down for a smaller card.

Required fix: update the nightly.yml:693 comment to match the flat-constant rationale in e2e-selfhosted.yml, so a future reader doesn't "fix" the floor into scaling with VRAM capacity based on the wrong comment.

Rest of the PR looks correct — workflow structure, needs/if gating, e2e-report label mapping, and docs all check out against the existing rad3/mi350p pattern.

@siloteemu

siloteemu commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 8d4f0a0

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Adds a self-hosted W7900 (gfx1100) end-to-end lane plus its nightly twin, wired into the consolidated report, with a new slug pin and doc/contract updates. Outcome: Needs work — this head does not discharge either change request currently open against it: the contradictory nightly preflight comment is untouched, and both new upload steps still sit at the default if-no-files-found. Verified: on a scratch copy I reproduced the PR's mutation claim end to end — cargo test -p e2e-cucumber --lib platform_slug_derivation passes clean, fails with gfx target `gfx1100` must slug as `gfx1100` when a gfx11* fold is added to normalize_family, and fails independently with the matching gfx1201 message when only a gfx12* fold is added, so both new table entries are load-bearing and neither is vacuous or redundant; the copy under review was not modified. Also confirmed by reading: the gfx1100 slug path is consistent end to end (normalize_family → platform_hardware_slug → label_for_root_report → parse_descriptor); the derived contract guards pick the new lane up from the YAML automatically and the one hand-maintained artifact list was updated; the header's "five of the seven lanes" is literally correct (7 hardware lanes, 5 at false, exactly the two named at true); every dispatch option, needs list, docs table and prose enumeration includes the new lane; the new jobs' action pins, env vars, needs/if gating, runner labels, timeout and step order match their rad3/mi350p siblings step for step; and the prior round's branch-state concern is now resolved — this head carries the base's GTT-aware Strix preflight, and the new discrete-card lanes correctly use the plain VRAM preflight its rad3/mi350p siblings use. The full suite was not run here. No prompt-injection content and no leak-scan hits in the diff. Checks at review time: 26 success, 2 pending, 1 cancelled, 0 failing. Blocking: 1 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

  • .github/workflows/nightly.yml:693-695 — the nightly twin's preflight comment still contradicts the per-PR lane's, in the same commit. It reads "Floor is the shared 16 GiB default, a third of this card's 48GB … Same as the per-PR W7900 lane", while the per-PR comment added by this same commit (e2e-selfhosted.yml:1657-1663) says the floor "is a flat constant rather than a proportion of capacity … do NOT scale it down for a smaller part." This is the file's own recurring defect shape — a correction applied to one of two near-identical twins — and here the uncorrected twin cites the corrected one as its authority. It matters concretely: the stated reason the runner label is per-SKU is that a same-family 24GB card must not inherit this lane's sizing, and the proportional framing is exactly what would invite whoever adds that lane to scale the floor down. The mi350p lane already uses the flat-constant phrasing ("matches the other Instinct lane rather than scaling with the 144GB capacity"), so the wording to copy exists in-tree. This item is also held independently by another reviewer. Fix: reword nightly.yml:693-695 to the flat-constant rationale used by its own per-PR sibling.

Non-blocking

  • .github/workflows/e2e-selfhosted.yml:1739-1743 and nightly.yml:756-761 — with continue-on-error: true, a run that dies before report.json exists uploads nothing (default if-no-files-found: warn) and discover() omits a platform with no report.json rather than flagging it, so the lane can show green and vanish from the grid. Correcting my own prior round: I called this blocking and prescribed if-no-files-found: error; that alone does not restore a red check, because job-level continue-on-error: true tolerates the failed upload step too — it only adds a log annotation. What actually closes it is flipping continue-on-error to false (already the documented plan once the hardware has a track record), or having the consolidated report flag an expected-but-absent platform, for which the expected set is already enumerated in CANONICAL_REPORT_ARTIFACTS. Same exposure as the shipped mi350p lane, and explicitly time-boxed by this PR's own docs change, so it is a known accepted gap rather than a new defect — but worth recording where the real fix lies.
  • .github/workflows/e2e-selfhosted.yml:1339-1349 — the WSL guard's comment asserts "Only the WSL lanes need this: every other lane writes report.json directly in-job", a premise that silently assumes the lane reports red on failure; it stops holding for a continue-on-error: true lane. A competent reader trips on this — one did during this review — so one clause ("…and fails red if it doesn't") would stop the conclusion recurring.
  • xtask/src/workflow_contract.rs:1717-1721 — the assertion message reads "self-hosted lane {} is continue-on-error, so the README must not document it as blocking", but the assertion fires for every self-hosted lane regardless of that value, and this PR's own header edit stresses the lanes are non-blocking because of branch protection, not continue-on-error. Pre-existing, but the message now contradicts the comment this commit just corrected.
  • .github/workflows/e2e-selfhosted.yml:1645 — the reclaim-step comment justifies itself with "The runner can serve several jobs in a row, and its work volume outlives any one of them", while the PR text describes this runner as scale-to-zero; the mi350p twin states both facts ("the runner scales to zero between jobs, but the work volume is not"). Carrying that phrasing over would keep the two consistent. Raised in the prior round and still unaddressed.
  • tests/e2e-cucumber/README.md:193-197 — the large-model paragraph still describes only the MI300X/vLLM and Strix Halo/Lemonade pairings; the workstation Lemonade lanes (rad3, and now this one) are absent. Pre-existing gap, but this PR is the second lane to fall outside the sentence. Also raised in the prior round and still unaddressed.

@siloteemu siloteemu left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 48bf12c

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Two things to fix before this lane is trustworthy.

The lane produces no failure signal for a whole class of failures. .github/workflows/e2e-selfhosted.yml:1345 and the nightly twin — with continue-on-error: true, a failure is tolerated in the Checks UI; and because the upload steps leave if-no-files-found at its default of warn, a run that dies before report.json exists (preflight failure, build failure, a crash before results are written) uploads nothing, and the consolidated report omits a platform with no report.json rather than flagging it. The lane then shows green and vanishes from the report — no row, no FAIL, nothing for a human to notice. This repo already wrote that hazard down in nightly.yml and guarded against it with if-no-files-found: error on the WSL copy-out step, but the guard was never extended to a GPU lane's own upload step; for the five lanes at continue-on-error: false the red check still covers it, and for this new lane nothing does. Adding if-no-files-found: error to both new upload steps closes it. Worth applying to the MI350P lane in the same pass — it has the identical exposure. For the record, assertion failures that do reach report.json are rendered truthfully as FAIL and are not masked; the hole is specifically pre-report failures.

The nightly twin's preflight comment contradicts the per-PR lane's, in the same commit. .github/workflows/nightly.yml:693-695 still reads "the shared 16 GiB default, a third of this card's 48GB … Same as the per-PR W7900 lane", while the per-PR comment added by this same commit says the floor "is a flat constant rather than a proportion of capacity … do NOT scale it down for a smaller part." Only one of the two instances was corrected, and the uncorrected one points at the corrected one as its authority. This matters concretely: the stated reason the runner label is per-SKU is that a same-family 24GB card must not inherit this lane's sizing, and the proportional framing is exactly what would invite someone adding that lane to scale the floor down.

The full findings, including four non-blocking items and a note on why a two-dot diff misreads this branch, are in the review comment on this PR.

@siloteemu
siloteemu dismissed their stale review September 29, 2026 15:08

Superseded — re-filing with the one item that still stands, plus a correction to a remedy in this request that does not actually work. See the new request.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 8d4f0a0

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

This supersedes my earlier change request on this pull request, which has been withdrawn. One item still stands, and one item in the old request carried a remedy of mine that does not work; both are set out below.

Still blocking

.github/workflows/nightly.yml:693-695 — the nightly twin's preflight comment contradicts the per-PR lane's, in the same commit. It reads "Floor is the shared 16 GiB default, a third of this card's 48GB … Same as the per-PR W7900 lane", while the per-PR comment added by this same commit (e2e-selfhosted.yml:1657-1663) says the floor "is a flat constant rather than a proportion of capacity … do NOT scale it down for a smaller part."

This is the twin shape this file keeps producing — a correction applied to one of two near-identical lanes — and here the uncorrected twin cites the corrected one as its authority. It matters concretely: the stated reason the runner label is per-SKU is that a same-family 24GB card must not inherit this lane's sizing, and the proportional framing is exactly what would invite whoever adds that lane to scale the floor down. The wording to copy already exists in-tree: the mi350p lane says "matches the other Instinct lane rather than scaling with the 144GB capacity".

Fix: reword nightly.yml:693-695 to the flat-constant rationale used by its own per-PR sibling.

Correcting my own earlier remedy

The previous request called the default if-no-files-found on the new upload steps blocking and prescribed if-no-files-found: error. That prescription does not close the gap: with job-level continue-on-error: true, a failed upload step is tolerated too and only produces a log annotation, so the check stays green either way. What actually closes it is flipping continue-on-error to false once the hardware has a track record — which this PR's own docs change already time-boxes — or having the consolidated report flag an expected-but-absent platform, for which the expected set is already enumerated in CANONICAL_REPORT_ARTIFACTS. Since the exposure is the same one the shipped mi350p lane already carries and is explicitly time-boxed here, it is a known accepted gap rather than a new defect, and it is no longer blocking. Apologies for the round-trip.

The remaining observations are in the review comment and none of them block.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Rebased onto main and addressed the review in two new commits, c903e3c and 2c8c5a8.

Blocking (@jussielo-amd, @siloteemu): the nightly W7900 preflight comment now uses the per-PR lane's flat-constant rationale. Both rad3 comments ("half of the R9700S's 32GB") had the same proportional framing and are corrected too.

Non-blocking, all taken:

  • Lane vanishes from the grid: both consolidated report jobs now fail, naming the lane, when a lane that ran uploaded no report. This covers every continue-on-error lane, including MI350P and all the nightly ones. Details are in the new last section of the description.
  • WSL guard comment: rewritten. It no longer says the report never flags an absent platform, and it no longer assumes the lane goes red on failure.
  • workflow_contract.rs assertion message: it now gives the real reason (no self-hosted lane is a required status check) instead of continue-on-error.
  • Reclaim comment: now matches the MI350P twin. The runner scales to zero, but the work volume persists.
  • Large-model paragraph: the model follows the lane's serve engine. MI300X and MI350P serve the vLLM model; Strix Halo, R9700 and W7900 serve the Lemonade checkpoint. The README, the feature comment and the step comment all say so now.

I've also corrected the description: it wrongly said Strix WSL2 failed on main at the old base.

@jussielo-amd

Copy link
Copy Markdown
Collaborator

Nice work here — this is a thorough, well-substantiated PR. The technical claims in the description (the Lemonade-vs-vLLM engine selection, the gfx1100/normalize_therock_family fallthrough risk, the 16 GiB floor rationale) all check out against the actual source, the new missing-report safety check is a genuinely useful addition, and the contract tests exercise the real embedded shell script rather than a reimplementation of it. Two rounds of self-review are visible in the commit history, and the lane passed on real hardware. Approving.

Findings

Blocking: none.

Verified against source (not just the diff):

  • rocm_core::normalize_therock_family (crates/rocm-core/src/lib.rs) does fold gfx1100 into gfx110X-all, while the harness's own normalize_family (capability.rs) has no such arm and falls through unchanged — confirms the "absence of a rule" risk this PR calls out, now pinned by a test.
  • family_prefers_vllm only allowlists *-dcgpu plus gfx906/908/90a, so gfx1100 correctly resolves to Lemonade, not vLLM — matches the PR's stated scope.
  • Blast radius: every file referencing the MI350P lane (closest precedent) has a matching W7900 addition; the nightly W7900 lane's structure matches the nightly MI350P lane exactly.
  • CI: all checks green, including E2E tests (W7900) itself.

Non-blocking / already-acknowledged tradeoffs:

  • This adds Lemonade coverage only, no vLLM exercise on this card — explicitly stated in the description and consistent with the R9700 precedent.
  • continue-on-error: true while the hardware is proven out — worth tracking as a follow-up to flip once it has a track record, not this PR's job.
  • The missing-report check can miss a lane on a re-run that uploads nothing but reuses an earlier attempt's artifact — documented limitation, never a false positive.

Positive signals:

  • The platform_slug_derivation test closes a pre-existing gap that shipped with the R9700 lane, not just covering the new one.
  • The workflow-contract tests execute the actual embedded shell script under bash -e with a stubbed gh, across ~10 scenarios (missing, skipped, cancelled-but-uploaded, absent-from-needs, API failure, etc.) — real behavioral coverage, not a Rust reimplementation that could drift.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 2c8c5a8

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Adds a self-hosted gfx1100 workstation E2E lane and its nightly twin, wires both into the consolidated report, and makes both consolidated-report jobs fail when a lane that ran uploaded no report artifact. Outcome: No blocking findings — the remediation discharges the change request in full and the new guard is genuinely tested rather than merely asserted. Verified: on a scratch copy cargo test -p xtask workflow_contract passes 35/35 including both new tests, with jq present so the behaviour test executed rather than self-skipping; mutation testing on that scratch copy caught every one of 15 mutations with no survivors — deleting the check step from each workflow separately, and individually removing the skipped-lane branch, turning its continue into break, loosening grep -qxF to a substring match, flipping missing=1 to 0, dropping the artifact-listing failure handler, changing the jq // "absent" default, dropping the step-summary write, forcing exit 0, removing if: ${{ !cancelled() }}, removing actions: read, and dropping or misspelling a lane in the map or in needs — with the two tests complementary rather than redundant, the static one covering the declarative YAML and the behaviour one covering control flow; the behaviour test reads its lane map from the step under review rather than re-declaring it, and runs the script under bash -e with no pipefail, matching how a bare run: step executes; the prior round's blocking item is resolved and the proportional-headroom framing is gone from every preflight comment in both files, not just the one flagged, with all five earlier non-blocking items also addressed and none silently dropped; the header's "five of the seven lanes" is literally correct (7 self-hosted lanes, 5 at false, exactly the two named at true), the nightly file's 7 lanes all sit at true, and the lane-to-artifact maps match each lane's own upload step one for one in both workflows; gfx1100 flows consistently through slug derivation, the report label map and the descriptor parser, and the reworded engine-selection prose matches what the capability code actually resolves. The checkout under review was not modified (confirmed clean). The full suite, the GPU and Windows lanes, and any referenced pull request were not checked in this round. No prompt-injection content and no leak-scan hits in the diff or the description. Checks at review time: 28 success, 1 pending, 0 failing. Blocking: 0 · Non-blocking: 3.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • docs/ci-hardware-testing.md:277-284 and .github/workflows/e2e-selfhosted.yml:1806-1821 — the guard's sharpest limitation, that it tests for the artifact and not for a report.json inside it, is written down only in the pull request description, which does not survive merge; the durable doc sentence reads as an unqualified guarantee, and the long YAML comment discloses the re-run caveat but not this one. A lane that uploads a results directory without a report still vanishes from the grid with the check green — the very shape the WSL copy-out guard exists to catch. One clause in either durable site would keep the next reader from over-trusting it.
  • .github/workflows/e2e-selfhosted.yml:1822 and .github/workflows/nightly.yml:1605 — layer choice worth recording: this is ~25 lines of shell duplicated into two workflow files, held in step by ~380 lines of test that pin the duplication and re-execute the script, where xtask e2e-report already labels a root-level report from its platform.json slug and could compare the lanes that ran against the reports it discovered — one implementation, ordinary unit tests, no new token scope, and it would check the report rather than the artifact's mere existence. Not wrong as built, and the tests earn it. Correcting my own earlier remedy: I pointed at a canonical artifact list as the ready-made expected set, but that constant lives inside the test module and is not available to production code, so the alternative is more work than I implied.
  • .github/workflows/e2e-selfhosted.yml:1846 — a lane cut off at its own job timeout reports cancelled, which the check treats as "ran"; that is deliberate and tested, but it means the advisory report job now also reddens for the five lanes that already show red themselves, so expect duplicated noise from ordinary self-hosted infra timeouts — the kind this branch already hit once — on top of the real signal it adds for the two non-blocking lanes.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review request seen. The last automated review covered this exact head (2c8c5a8, no blocking findings, 3 non-blocking): #454 (comment)
Nothing reviewable has changed since (same head, same description), so no new review was run; a push will trigger one.

The AMD Radeon PRO W7900 (gfx1100, 48GB) had no E2E coverage at any
cadence. This adds the per-PR lane and its nightly twin, wired into the
consolidated report.

The runner label is `w7900` rather than a `gfx1100` family label. The
RX 7900 XTX shares that ISA but carries half the VRAM, and GitHub
dispatches a job to any runner whose labels are a superset of the
job's, so a family label would send 48GB-sized work to a 24GB card --
the mismatch the per-hardware label convention exists to prevent.

The preflight floor stays at the shared 16 GiB default. That value is a
flat constant, not a proportion of VRAM: e2e-gpu and e2e-gpu-mi350p use
the same 16 GiB against very different capacities, because the floor
only has to catch a leftover serve while letting a clean host pass on
the first poll.

Both lanes are non-blocking while the hardware is proven out, matching
how the MI350P lane was introduced.

`gfx1100` slugs as itself through `normalize_family` and
`platform_hardware_slug`, exactly as `gfx1201` does for the R9700 lane,
so the report needs only the artifact label mapping. That fallthrough is
the absence of a rule, so `platform_slug_derivation` now pins both
families: `rocm_core::normalize_therock_family` folds them into
`gfx110X-all`/`gfx120X-all` for wheel selection, and aligning the two
functions would route both lanes' artifacts to `e2e-unknown-report`
with every test still green.

Note the coverage this adds is Lemonade-shaped: `family_prefers_vllm`
gates on data-center families, so `gfx1100` resolves to lemonade and the
`@requires-engine:vllm` scenarios skip -- the same profile the shipped
R9700 lane has.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The nightly W7900 preflight described the 16 GiB floor as a third of
the card's 48GB, while its per-PR twin says the floor is a flat
constant that must not scale with capacity. The proportional wording is
exactly what would invite whoever adds a same-family 24GB lane to lower
the floor, so it now carries the per-PR rationale. Both rad3 preflight
comments framed the same floor as half of the card's 32GB and get the
same correction.

The per-PR reclaim comment now says what the MI350P twin says: the
runner scales to zero between jobs but its work volume persists.

The contract test's message tied "not blocking" to continue-on-error,
contradicting the workflow header; it now names the real reason, that
no self-hosted lane is a required status check.

The large-model docs named only MI300X and Strix Halo. The model
actually follows the lane's serve engine, so MI350P gets the vLLM model
and R9700 and W7900 get the Lemonade checkpoint; the README, the
feature comment and the step comment now say so.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
A lane that dies before writing report.json (preflight, build, a crash
before results exist) uploads nothing, and the consolidated grid can
only render reports that arrived, so the lane dropped out with no row
and no FAIL. On a continue-on-error lane -- the W7900 and MI350P per-PR
lanes and every nightly lane -- nothing else turned red either, so the
failure was invisible.

Both report jobs now check that every lane that ran (any `needs` result
but skipped) uploaded its report artifact, reading the run's artifact
list rather than the download directory, which flattens a single match
and loses its name. A missing report fails the job with the lane named
in an annotation and the step summary; an API failure says so rather
than dying without a word. The check can miss a re-run that uploads
nothing, because earlier attempts' artifacts stay listed, but it never
flags a lane that did upload.

The contract tests pin the lane-to-artifact map against each lane's
upload step, the step's guard and inputs, and the per-PR and nightly
copies of the script against each other. A behaviour test runs the
script under `bash -e` with a stub `gh` across every lane reported, a
lane that ran and uploaded nothing (including as success or cancelled),
a skipped lane, lanes after a skipped one, two lanes missing at once, a
lane absent from `needs`, a timed-out lane that still uploaded, a
near-miss name and an API failure; on CI a missing `jq` fails it rather
than skipping.

The WSL upload guard's comment is rewritten to match, since its claim
that the report never flags an absent platform no longer holds.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Main moved under this branch: per-PR rad3/mi350p lanes were removed, the
GPU reclaim became scripts/reclaim-gpu.sh, preflights poll at least once
before their deadline and report holders, and the nightly lanes and the
strix-ubuntu lane gained a release/nightly channel matrix.

Apply the same to both W7900 lanes, and teach the missing-report check to
see a channel-matrix lane through its release leg, since the report job has
no matrix context.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit 9f1c97e Oct 15, 2026
42 of 43 checks passed
@tomastola
tomastola deleted the ci/e2e-w7900-lane branch October 15, 2026 00:10
juhovainio added a commit that referenced this pull request Oct 15, 2026
Rebase picks up #454's W7900 lane (PR #415/#454/#463 all now merged).
Drop the hardcoded '4 jobs'/'4 platforms' counts from the validation
ladder table so they can't drift again as lanes are added, and credit
ci_yml_schedules_no_self_hosted_job alongside
self_hosted_workflow_owns_the_gpu_lanes for what each test actually
checks.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
jussielo-amd pushed a commit to jussielo-amd/rocm-cli that referenced this pull request Oct 16, 2026
…Cm#465)

* docs(ci): document the three-stage validation ladder

Add the missing framing to docs/ci-hardware-testing.md: the three
self-hosted GPU gates (per-PR smoke, nightly coverage, release-candidate
regression) and how they differ; the validation axes (OS, hardware,
channel, SDK-version); why the serve engine isn't an independently
selectable matrix axis (quotes effective_serve_engine()); and a short
addendum on e2e-prewarm's --version/--build-date pin flags. No lane,
artifact, or job-order changes, so every contract-tested table/list/
sentence in the doc is left byte-identical.

ROCMAI-431

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci): fix e2e-prewarm tense, reword two overstated doc claims

Review found the --version/--build-date paragraph claiming present-tense
availability for a flag pair that isn't in this tree yet (ships in ROCm#464,
stacked on ROCm#415, neither merged) — contradicted the ladder section's own
"ships in ROCm#464" framing two paragraphs up.

- e2e-prewarm --version/--build-date: reworded to future tense, matching
  the ladder section.
- Per-PR row: note e2e-gpu-strix-ubuntu already runs both channel legs
  per PR, not just 4 flat jobs.
- "cannot be silently undone": reworded to match what
  self_hosted_workflow_owns_the_gpu_lanes actually checks (job keys
  present as text, not a structural pin).

Left the two open-decisions-for-the-author items untouched per review
(non-defects).

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci): reword ROCm#415/ROCm#464 merge-status passages now that ROCm#415 landed

PR ROCm#415's release/** trigger merged 2026-10-01; e2e-selfhosted.yml on
main already triggers on push: branches: [main, "release/**"]. Only
ROCm#464 (the SDK pin) is still pending. Fixes the two passages in the
ladder section and the pre-warm section that still said neither PR
was merged.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci): fix engine-selection doc errors, de-stale the ROCm#464 status prose

pr-review-watcher flagged 3 blocking issues at f679385:
- the engine section said rocm serve picks its engine via the harness's
  effective_serve_engine(), when capability.rs's own header says that
  function is the re-implementation and select_serve_engine (main.rs)
  is the product original.
- "the engine is not selectable independent of hardware and OS"
  overstated it: select_serve_engine checks an explicit --engine and a
  configured default_engine before falling back to the GPU-family
  preference, and a scenario can pin either side with
  @requires-engine:<vllm|lemonade>.
- the ROCm#464 merge-status prose (two copies) asserted "still pending as
  of this writing", which goes false the moment ROCm#464 merges. Reworded
  both to state what the tree supports today instead of PR status.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci): rebase onto main, fix stale citations and self-contradiction

- rebase onto main (PR ROCm#415/ROCm#460/ROCm#463 already merged there)
- cite select_serve_engine's real location (serve_cmd.rs, not main.rs) and
  turn the three new bare-backtick file citations into relative markdown
  links per AGENTS.md's citation rule
- drop the sentence implying e2e-prewarm already has --version/--build-date
  flags; state once (in the ladder section) that the pin is pending work
  (ROCMAI-430) and cross-reference it from the pre-warm addendum instead of
  repeating it
- describe select_serve_engine's full precedence (recipe-support gate,
  recipe's own preferred engine, platform default), not just the two-step
  simplification
- say 'native Windows' for the lemonade-only engine case; WSL reports
  os_family linux and stays vLLM-eligible

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci-hardware-testing): link the therock.rs citation per AGENTS.md §5

A bare backtick path doesn't get checked by docs-links, so it can go
stale silently. Link it relative to this file, and name the actual
private function instead of just the file.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci): rebase onto main and fix stale lane counts and test credit

Rebase picks up ROCm#454's W7900 lane (PR ROCm#415/ROCm#454/ROCm#463 all now merged).
Drop the hardcoded '4 jobs'/'4 platforms' counts from the validation
ladder table so they can't drift again as lanes are added, and credit
ci_yml_schedules_no_self_hosted_job alongside
self_hosted_workflow_owns_the_gpu_lanes for what each test actually
checks.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

* docs(ci): trim the duplicated e2e-prewarm pin-status explanation

The pre-warm section restated the no-pin/ROCMAI-430 fact that the
validation-ladder section already explains, contradicting the PR's own
claim that it's stated once. Cross-reference instead of repeating it.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

---------

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants