Repository navigation
docs(ci): document the three-stage validation ladder (ROCMAI-431) - #465
Conversation
46f87e6 to
930f816
Compare
tomastola
left a comment
There was a problem hiding this comment.
What the change does. Docs-only: adds a "three-stage validation ladder" section to docs/ci-hardware-testing.md (per-PR, nightly, release-candidate gates), a section on why the serve engine is not a matrix axis, and notes on the (platform, channel) report columns and the e2e-prewarm pin flags. I checked the new claims against the code at this head. The workflow triggers, lane names, nightly cron and channel matrices, effective_serve_engine behaviour and report column keying are accurate. Two claims are not (below).
Coverage. Single full read of the PR's own diff (one file), checked against the workflows, xtask and capability.rs at head 930f816. This head is a rebase of the previously reviewed 46f87e6; the PR's own diff is byte-identical, and the parent commits that moved touch none of the code this doc describes. I did not read the earlier discussion before forming the findings and did not run any tests.
Needs fixing before merge
- The
e2e-prewarmparagraph (inline) describes--version/--build-datein the present tense, but that code is not in this tree, and the ladder section of the same doc says it ships in #464. Someone following the prewarm section would pass a flag that does not exist.
Non-blocking
- The per-PR row says it covers the 4 platform jobs, but
e2e-gpu-strix-ubuntualready runs a[release, nightly]channel matrix per PR (inline). - "cannot be silently undone" overstates what
self_hosted_workflow_owns_the_gpu_laneschecks (inline).
Open decisions for the author (not defects)
- The
effective_serve_enginesource is pasted verbatim into the doc. It reads well, but it drops the source's WSL comment and will drift silently whencapability.rschanges. Referencing the function by name, as the rest of the doc does, would age better. Deliberate? - The paragraph hard-codes #415 and #464 and "as of this writing". It goes stale once either merges. A ticket reference, or a marker whoever merges them will find, is easier to keep true.
The merge decision is yours; this review is posted as a comment and files no approval or change request.
| - prunes with `rocm storage remove-old-installs` after any install, update, or | ||
| repair, so the multi-version cache stays bounded. | ||
|
|
||
| `e2e-prewarm` also accepts mutually exclusive `--version`/`--build-date` |
There was a problem hiding this comment.
The flags described here do not exist yet. This paragraph says e2e-prewarm "also accepts mutually exclusive --version/--build-date flags" and that "today no caller in this repo passes either flag yet". There is no --version/--build-date handling under xtask/src at this head, and the ladder section of this same doc (lines 43-49) says they ship in #464, which is open and stacked on the also-open #415. The code and the ladder paragraph are the accurate side; this paragraph is not. Reword it as future/conditional ("#464 adds…"), drop it in favour of the single mention in the ladder section, or land this PR after #464.
There was a problem hiding this comment.
Resolved at 201409b: the pre-warm paragraph now says the flags do not exist yet and points at ROCMAI-430.
|
|
||
| | Rung | Workflow | Trigger | Lanes | Blocks a merge/tag? | | ||
| |---|---|---|---|---| | ||
| | Per-PR smoke gate | `e2e-selfhosted.yml` | `push`/`pull_request`/`merge_group` (see "Triggers") | The 4 jobs in the Platforms table below | No — absent from the required-status-check list | |
There was a problem hiding this comment.
Per-PR row is incomplete (non-blocking). "The 4 jobs in the Platforms table" leaves out that e2e-gpu-strix-ubuntu in e2e-selfhosted.yml already has strategy.matrix.channel: [release, nightly] and uploads e2e-gpu-strix-ubuntu-${{ matrix.channel }}-report (from the ROCMAI-125 parent commit). So the per-PR gate is really 5 job runs and is not release-channel-only; the channel matrix and the two-column report (line 177) are not nightly-only. Worth stating in the row.
There was a problem hiding this comment.
Fixed in 6534052 — the row now states e2e-gpu-strix-ubuntu runs both its [release, nightly] channel legs per PR (ROCMAI-125).
There was a problem hiding this comment.
Resolved at 201409b: the per-PR row now states the e2e-gpu-strix-ubuntu [release, nightly] legs. (The count in that row is a new issue; see the review.)
| thus their own concurrency group — means an offline runner can only ever stall | ||
| that workflow's own supersession, never `ci.yml`'s required checks. See | ||
| `EAI-7548`. | ||
| `EAI-7548`. `xtask/src/workflow_contract.rs`'s |
There was a problem hiding this comment.
"cannot be silently undone" overstates the test (non-blocking). self_hosted_workflow_owns_the_gpu_lanes (xtask/src/workflow_contract.rs) only asserts sh.contains(job) for the three job keys, a substring check that any mention in a comment satisfies. It also does not cover e2e-wsl, which a separate test handles. Suggest "requires the three job keys to appear in the file".
There was a problem hiding this comment.
Fixed in 6534052 — replaced with your suggested wording: the test requires the three job keys to appear in e2e-selfhosted.yml, so moving one back to ci.yml fails CI rather than merging unnoticed.
930f816 to
a01b7b8
Compare
|
Addressed the review in
Left the two open-decisions-for-the-author items (the pasted
|
a01b7b8 to
272a672
Compare
272a672 to
6534052
Compare
6534052 to
3b59f96
Compare
3b59f96 to
77d03ef
Compare
77d03ef to
6557bc8
Compare
jussielo-amd
left a comment
There was a problem hiding this comment.
Nice work on this one — the ladder table and the effective_serve_engine writeup are exactly the kind of grounding this doc needed, and the level of cross-checking against the actual YAML/Rust (down to verifying the "suspected stale" artifact-naming paragraph was actually still correct) is the right way to write CI docs. I verified the quoted effective_serve_engine/family_prefers_vllm logic, the nightly 6-platform/channel-matrix claims, the self_hosted_workflow_owns_the_gpu_lanes test description, and the no-duplicate-lane-claims grep — all check out against current main, and cargo fmt/cargo test -p xtask --all-targets are clean.
One thing needs fixing before merge though: PR #415 (the release/** trigger) already merged on 2026-10-01, and e2e-selfhosted.yml on main already has push: branches: [main, "release/**"]. The doc currently says (twice — in the ladder section and in the pre-warm section) that "neither [#415 nor #464] is merged... e2e-selfhosted.yml on main today still triggers on push: branches: [main] only." That was true when you wrote it (Sep 30) but is now stale relative to main — the branch just needs a rebase and those two passages reworded to reflect that #415 merged and only #464 (the SDK pin) is still pending.
Everything else here looks solid.
6557bc8 to
f679385
Compare
|
Addressed the review in |
jussielo-amd
left a comment
There was a problem hiding this comment.
Independently re-verified every factual claim in this doc against source at head (f679385): workflow_contract.rs's job-key test, e2e-selfhosted.yml's push branches, nightly.yml's cron/channel matrix across all 6 platforms, capability.rs's effective_serve_engine logic, and therock.rs's pub(crate) visibility. All check out, and #415/#464 merge-status wording is now accurate (confirmed #415 merged, #464 still open).
The staleness issue I flagged in my prior CHANGES_REQUESTED review was fixed in f679385. No remaining blocking issues — approving.
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · f679385
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change (docs-only: docs/ci-hardware-testing.md).
The description says "documentation only" and lists five additions, and all five are present. The title cites a ticket ID, which was not retrieved.
All three findings below are new this round (first review of this PR).
Blocking
-
The engine section says the wrong code makes
rocm servepick its engine —docs/ci-hardware-testing.md:211-214
The doc saysrocm serve"picks it viaeffective_serve_engine()intests/e2e-cucumber/src/capability.rs". The product selects the engine inselect_serve_engine(apps/rocm/src/main.rs:5930), throughpreferred_serve_engine_for_host_gpu_summary.effective_serve_engineis the harness's own re-implementation.capability.rs:13-15says it is "RE-IMPLEMENTED here … duplicating the product'sselect_serve_engine". The "(mirroring …)" aside reverses which side is the copy. The product governs because the harness's own header calls itself the duplicate.
Confidence 90 · mechanical · Fix: say the harness'seffective_serve_engine()predicts whatrocm servepicks with no--engine. -
"The engine is not selectable independent of hardware and OS" overstates the product —
docs/ci-hardware-testing.md:210-211,:238-239
select_serve_engine(main.rs:5936-5948) takes an explicit--enginefirst and a configureddefault_enginesecond. The GPU preference only applies when neither is set, and then only if the model's recipe supports that engine (main.rs:5950-5956).capability.rs:37itself says the helper models "whatrocm serve <model>would pick with no--engine". The conclusion for the lanes holds: no workflow passes an engine, and the harness skips a vLLM@requires-enginepin on hosts that cannot run it. The stated reason is wrong, and "Adding a vLLM lane means adding hardware" is only true of the harness's default path. The code governs.
Confidence 85 · logic · Fix: narrow the claim to the E2E harness's default engine, and mention the explicit/configured override and the@requires-enginepins. -
Doc states the open status of PR #464 and will go false once it merges —
docs/ci-hardware-testing.md:44-56,:349-354
"still pending as of this writing", "will also accept", "what every lane in this tree runs today" and "not part of either PR" are all true only while #464 is open. At head,xtask/src/main.rs:219-231has no--version/--build-date. The PR's own third commit had to reword the #415 status for the same reason. The same status is also written twice (lines 46-56 and 349-354), so one merge makes two passages stale. No test constrains this prose. The other time-bound statement in this doc is dated ("confirmed 2026-09-11"), and AGENTS.md §4 says docs should not carry scratch or planning state.
Confidence 80 · logic · Fix: describe only what the tree supports today ("e2e-prewarmhas no version pin"), or date the statement and drop one of the two copies. Keep the PR dependency in the PR body.
Non-blocking
- The
effective_serve_enginecode block is presented as a quote but is not exact —docs/ci-hardware-testing.md:216-227. It drops the two comment lines atcapability.rs:47-48. The block is also an unguarded copy: no xtask test reads code blocks in this doc.capability.rs:44-45already plans to replace this function, so the copy is likely to go stale. The prose at lines 229-231 already carries the rule. Confidence 75 · mechanical · Fix: drop the block, or mark it as abbreviated. - "The 4 jobs in the Platforms table" is ambiguous —
:36,:38. That table has 5 rows including the mocke2ejob fromci.yml. The four are thee2e-selfhosted.ymljobs. Confidence 75 · mechanical · Fix: "the 4e2e-selfhosted.ymljobs". self_hosted_workflow_owns_the_gpu_lanesis described more strongly than it checks —:22-26.xtask/src/workflow_contract.rs:755-773checks that the stringse2e-gpu:etc. appear ine2e-selfhosted.yml, not that they are job keys. A lane moved back intoci.ymlwould still fail it. Confidence 70 · mechanical · Fix: "must appear ine2e-selfhosted.yml".- New text repeats existing text in the same doc —
:188-191(existing:197-203already says release and nightly runs get separate columns),:36(repeats:74-79),:40-42(repeats:184-185). Each is a maintenance cost, not a defect. Confidence 70 · mechanical · Fix: keep one copy of each. - Stray line wrap —
:51-53("not part of either\nPR: nothing in this\nrepo maps") looks like an editing leftover. Confidence 70 · mechanical · Fix: reflow.
Decisions for the author
- A ladder table with job names, cron time and lane counts that no test guards — tradeoff
The facts check out against the workflows at head, including the 06:00 UTC cron, thee2e-gpu-nightly-rad3/-mi350pnames, the channel matrices, therelease/**push trigger and the(platform_slug, channel)column keying. The doc-contract test parses only the| Job | Workflow | Platform | Runner labels |table, so none of this will fail CI if it drifts. One side: this is the framing the doc was missing. The other: it adds drift-prone counts. Confirm the maintenance cost is acceptable. - "None are required-status checks" — tradeoff
Branch protection is not visible from the repo, so this could not be verified here. It is consistent with the header ofe2e-selfhosted.ymland the existing "Blocking vs. non-blocking" section. Confirm it is still true.
Positive signals
- The doc states plainly that no
sdk_version: [current, n-1, n-2]matrix exists and why. That avoids implying a mechanism that is not there. - Every ladder-table claim about triggers, lanes and channels matched the workflows at head.
Deployment notes
None
What this covered
Every added line of docs/ci-hardware-testing.md and the doc in full, against prw-base...f6793857 (3 commits, one file). Claims were checked against .github/workflows/{e2e-selfhosted,nightly,release}.yml, xtask/src/{workflow_contract,main,e2e_prewarm}.rs, tests/e2e-cucumber/src/capability.rs, apps/rocm/src/main.rs, crates/rocm-core/src/lib.rs, apps/rocm/src/therock.rs and crates/e2e-report/src/consolidated.rs. Three read-only workers covered the code-claims, engine/prewarm claims and the instruction/history/comment passes. The engine-selection and capability.rs findings were re-checked against source. Nothing was built or run. Impact analysis was a docs-only grep of other docs and the xtask doc-contract tests.
- Checks at this head: all reported checks succeeded. These did not report (skipped, so they gave no signal):
E2E tests (MI300X),E2E tests (Strix Halo, Ubuntu),E2E tests (Strix Halo, Windows),E2E tests (Strix Halo, WSL2),E2E consolidated report,E2E consolidated report (self-hosted),Skill checks (skillscope),Sphinx docs build (-W),Build Linux PR binariesandremote control channel (containerised). The Sphinx-Wdocs build did not run, so docs build warnings from this file are unchecked. - Did not run: the prior-changes pass (earlier reviews are not read, by rule). Ticket references and the branch-protection claim could not be verified from the repo.
- Not reconciled against any discussion here.
|
Addressed the review in |
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 04234c7
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change, 4 commits, one file: docs/ci-hardware-testing.md. The doc was also checked against the current base tip (prw-base), which is many commits ahead of the merge-base, because it lands on top of that.
The PR description says it is stacked on #463 ("Depends on #463"). It also cites ROCMAI-431, ROCMAI-125, ROCMAI-429, ROCMAI-430, ROCMAI-120 and EAI-8761. None of those tickets could be retrieved from here.
Round 2. Each finding is marked new (first raised this round) or standing (raised in the round at f679385 and still holding). Of the three blockers filed at f679385, two no longer hold and are dropped: the engine-selector direction and the "not selectable independent of hardware" overstatement. Neither was re-found by this round's blind review, and each was then put to the reviewer, which found the reworded engine section resolves it. The third (status prose) still holds and is carried below as standing.
Blocking
-
[new] The PR description no longer matches the doc — PR body vs
docs/ci-hardware-testing.md:44-56,:355-359,:219-230- The body says the ladder section "States plainly that the release-candidate rung's trigger ships in PR #415 and its SDK-version pin flags in PR #464, both stacked but not yet merged".
- The doc now says the opposite for #415 ("merged in PR #415, so
e2e-selfhosted.ymlonmainalready triggers on…") and does not mention #464 at all. - The body says it quotes
effective_serve_engine()"verbatim". The doc's block drops the two comment lines inside the function (capability.rs: "The vLLM adapter bails on native Windows (WSL builds as Linux, …)"), so it is not verbatim. - The body also says the addendum describes "
e2e-prewarm's--version/--build-datepin flags". The addendum (:355-359) says those flags do not exist at HEAD and that another ticket "adds" them. - The description claims things that are not in the change. Remedy: rewrite the description to match the doc as it now stands.
- Confidence 85 · logic · Fix: update the body.
-
[new] The doc contradicts itself about whether
--version/--build-dateexist —docs/ci-hardware-testing.md:53-56vs:46-47,:355:53-56says "the release gate's SDK version is whatever--version/--build-dateits caller passes".:46-47and:355saye2e-prewarmhas no such flags.- The tree supports the second reading.
git grep -n "build-date\|build_date" prw-base -- xtaskreturns nothing. TheE2ePrewarmclap enum has onlychannel,keepandprewarm_dir. - Every lane calls
e2e-prewarm --channel … --prewarm-dir …with no pin. On arelease/**push the SDK is therefore whatever the channel index currently serves. No caller exists that could pass a pin. - The "whatever its caller passes" clause governs nothing: it describes a future state in the present tense.
- Confidence 85 · logic · Fix: say the release gate currently tracks the channel index's latest build. Describe the pin as future work, or drop that clause.
-
[new] A new source citation points at the wrong file on the current base, and the new citations break an AGENTS.md rule —
docs/ci-hardware-testing.md:213,:22-23,:215- Line 213 says
select_serve_engineis inapps/rocm/src/main.rs. That is true at the PR head (main.rs:5930). - On
prw-baseit is not:git grep -n "fn select_serve_engine" prw-base -- apps/gives onlyapps/rocm/src/serve_cmd.rs:62.main.rsjust re-imports it (use crate::serve_cmd::{…, select_serve_engine, …}). prw-base:AGENTS.md:182-199(§5) says: "cite a specific repo file in markdown as a link relative to the citing file — … not as a bare backtick path: thedocs-linksCI job and thelycheeprek hook fail when a link's target is moved or deleted, but cannot see a bare path go stale." It covers "citations you add or edit".docs/ci-hardware-testing.mdis not in its exclusion list.- The PR adds bare backtick citations of
xtask/src/workflow_contract.rs(:22-23),apps/rocm/src/main.rs(:213) andtests/e2e-cucumber/src/capability.rs(:215). The stale:213path is the failure the rule exists to prevent. - The base's rule governs, because the doc lands on top of it.
- Confidence 90 · mechanical · Fix: use relative links such as
[serve_cmd.rs](../apps/rocm/src/serve_cmd.rs), name the symbol rather than a line, and point atserve_cmd.rs.
- Line 213 says
-
[standing] The doc states the open status of the SDK-pin work and will go false once it lands —
docs/ci-hardware-testing.md:44-56,:355-359- This was raised at f679385 against wording that named PR #464. The rewording replaced the PR number with a ticket ID and "at HEAD", but the passages are still status: "ROCMAI-430 adds mutually exclusive flags for that" (
:46-48) and "ROCMAI-430 adds …" (:357) describe, in the present tense, code that is not in the tree. "a separate, not-yet-started step" (:51) and "what every lane in this tree runs today" (:356) are also true only until that work merges. - The same fact is still written twice (
:46-50and:355-359), each copy pointing at the other, so one merge makes both stale. - The "no pin" clause on its own does describe the tree today and is fine.
- This round's blind review did not re-find it as worded; it was settled by putting the earlier objection to the reviewer after its report was returned, which found that it still holds. The blind review independently filed the sharpest instance (
:53-56) above and the status wording as a non-blocking item below. - Confidence 80 · logic · Fix: describe only what the tree supports today ("
e2e-prewarmhas no version pin; release-branch runs track the channel index"), state it once, and keep plans for the pin work in the PR or issue that adds it.
- This was raised at f679385 against wording that named PR #464. The rewording replaced the PR number with a ticket ID and "at HEAD", but the passages are still status: "ROCMAI-430 adds mutually exclusive flags for that" (
Non-blocking
-
[standing] The
workflow_contractparagraph overstates the test —docs/ci-hardware-testing.md:22-26- The test at
prw-base:xtask/src/workflow_contract.rs:755requires only that the raw text ofe2e-selfhosted.ymlcontainse2e-gpu:,e2e-gpu-strix-ubuntu:ande2e-gpu-strix-windows:. It does not parse job keys. - The guard that fails a PR moving a lane back into
ci.ymlis the separate test at:735,ci_yml_schedules_no_self_hosted_job. It rejects self-hosted labels inci.yml'sruns-on. The doc does not name it. - Confidence 70 · logic · Fix: name both tests, or say "contains".
- The test at
-
[standing] Hard-coded lane counts, and one is ambiguous —
:36,:37,:38- "The 4 jobs in the Platforms table" is wrong against that table, which has 5 rows (
e2eis the GitHub-hosted mock job). Four is right only for the self-hosted jobs. - Nothing guards these counts, unlike the lane lists in
hardware_testing_docs_cover_all_self_hosted_platforms. - Confidence 70 · mechanical · Fix: say "the four self-hosted jobs" or name them.
- "The 4 jobs in the Platforms table" is wrong against that table, which has 5 rows (
-
[new] The engine section understates how
rocm servechooses an engine —:210-217,:232-245- The doc says explicit and configured overrides come first, then "falls back to the GPU-family preference". The product (
serve_cmd.rs:62-106onprw-base) also requires the recipe to support the preferred engine, then falls through to the recipe's first preferred engine, then the platform default. - "Any Windows host resolves to
lemonade" should say native Windows. The source comment the doc dropped says WSL stays vLLM-eligible. - The harness's
normalize_familyis a subset of the product'snormalize_therock_family. For example it does not mapgfx125*or the genericgfx90*prefix to the-dcgpufamilies. The advice "add hardware from a vLLM-eligible family" is reliable only for the families the harness maps. - Confidence 70 · logic · Fix: soften the "what
rocm servewould pick" framing and say native Windows. Linkcapability.rsinstead of restating its rule.
- The doc says explicit and configured overrides come first, then "falls back to the GPU-family preference". The product (
-
[standing] The pasted
effective_serve_engineblock —:219-230- It is copied code that already exists in the repo. The block calls
family_prefers_vllm, which is not shown, and nothing keeps the copy in sync. Only link resolution is checked in this repo, not snippet content. - Confidence 80 · mechanical · Fix: link the symbol and keep the prose rule.
- It is copied code that already exists in the repo. The block calls
-
[new] Status and time-capsule prose —
:44,:47,:51,:355-359- "merged in PR #415", "at HEAD", "not-yet-started" and "ROCMAI-430 adds…" are status wording that goes stale as soon as the pin flags land.
- Confidence 78 · mechanical · Fix: describe the current unpinned behaviour only. Document the flags in the change that adds them.
-
[standing] Repetition —
:46-50vs:355-359- The "no pin at HEAD / ROCMAI-430" fact is stated twice, with cross-references between the copies. The R9700/MI350P demotion also appears at
:40-42and again at:184-185. - Confidence 65 · mechanical · Fix: state each once and link.
- The "no pin at HEAD / ROCMAI-430" fact is stated twice, with cross-references between the copies. The R9700/MI350P demotion also appears at
-
[new] Nightly lane names disagree within the doc —
:37vs:184,:193-199- The new ladder table uses
e2e-gpu-nightly-rad3ande2e-gpu-nightly-mi350p. These are the job keys innightly.yml(:490and:588at HEAD). - Pre-existing wording at
:184says nightly "also runse2e-gpu-rad3ande2e-gpu-mi350p". Those are artifact and slug names, not job keys. The PR edits that same paragraph without reconciling the two. - The table's names govern, because they match the workflow job keys.
- Confidence 70 · mechanical · Fix: reword
:184. This is a pre-existing wording connected by pattern.
- The new ladder table uses
-
[new] Commit structure —
83a1ce1,498b180,f679385,04234c7- Three follow-up commits re-edit passages from the first, and
f679385reverses498b180's framing. This is review-fixup noise on a single logical change. 83a1ce1's body says it adds "the validation axes (OS, hardware, channel, SDK-version)", but the doc has no such section.- Confidence 70 · mechanical · Fix: squash, or reword the message.
- Three follow-up commits re-edit passages from the first, and
Decisions for the author
-
Copy the rule into the doc, or link it — tradeoff
- A pasted snippet reads well at a glance. A link cannot drift silently and matches the base's citation rule. Moving to a link is a call for the author to make.
-
Stacked on #463 — tradeoff
Positive signals
- Almost every checkable claim in the added text is accurate:
- the report's
(platform_slug, channel)column keying (crates/e2e-report/src/consolidated.rs:527-541); - the nightly cron
0 6 * * *; - the
[release, nightly]matrix on the nightly lanes and one2e-gpu-strix-ubuntu; - the
release/**push trigger; - the absence of any n-1/n-2 mapping;
- the private
therock.rsindex parsers; - the
@requires-engine:<vllm|lemonade>spelling; - the vLLM-preferred family list.
- the report's
- No lane, artifact or job-order text that
hardware_testing_docs_cover_all_self_hosted_platformskeys on was touched. - Docs-only scope is stated in the PR body, so no Gherkin scenario is owed.
Deployment notes
None
What this covered
- Read: the whole of
docs/ci-hardware-testing.mdat the PR head, and the full diffprw-base…HEAD(4ee61616…04234c79). Cited claims were checked against code at both the PR head andprw-base(34fbdfe). A three-way merge of this file againstprw-basewas reported conflict-free by a worker. - Fan-out: three read-only workers: ladder and pre-warm claims; engine section and the harness against the product; and the source-reading passes (AGENTS.md adherence, history, comments, doc quality, commit messages). Each ran to completion. Their results were then re-checked directly, namely:
- the
serve_cmd.rs:62location; - AGENTS.md §5 at
prw-base; - the two
workflow_contract.rstests; - the dropped comment lines in
capability.rs; - the absence of any
build-dateflag inxtask; - the PR description.
- the
- Not run: history and prior-changes passes ran only against file history. No prior reviews were read during the review, by design. No cargo build or tests were run (docs only, so the cargo gates are not exercised by this change).
- Standing objections: settled after the blind review was returned, by putting each one to the reviewer with the author's reply. Those verdicts are anchored evidence (the reviewer was told what to look for), and they did not add, drop or change any finding above.
- Checks that could not be made:
- The cited ticket IDs were not retrieved.
- The branch-protection required-check list lives in GitHub settings. The "none are required" claim rests only on the dated comment at
e2e-selfhosted.yml:11-17.
- CI: all reported checks passed on this head. These had not reported: E2E Strix Halo Ubuntu, E2E Strix Halo Windows, E2E Strix Halo WSL2, E2E MI300X, E2E consolidated reports, Sphinx docs build, Skill checks, remote control channel and Build Linux PR binaries. For a docs-only change the Sphinx docs build is the one whose absence matters. That class is not evidenced here.
- Not reconciled against the review discussion beyond the standing objections above.
04234c7 to
a7f1121
Compare
|
Addressed the review (all three blocking findings, across
Also fixed in the final pass: the bare-backtick-path citations are now relative markdown links per AGENTS.md §5, and the "native Windows" vs. WSL lemonade-only engine case is corrected (WSL reports Non-blocking items (ambiguous "4 jobs" phrasing, repeated text, stray line wrap, abbreviated code-block quote) were also cleaned up in the same passes.
|
Superseded by round-2 review and the subsequent fix commits; round-1 blockers (engine-selector direction, overstated hardware/OS claim) confirmed resolved in round 2.
|
Round 2 (review 5488244550) addressed in
Also rebased onto current Round 1 (5429469170) and round 2 (5488244550) reviews dismissed as resolved; re-requesting review. |
tomastola
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · a7f1121
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Follow-up review — one blocking item, one-line fix
Re-review of the new commits since my 2026-09-30 review (full review of the whole change, one docs file). Filed as a comment rather than a change request: the item below is a link-style fix with no effect on what the doc says.
Earlier findings (my 2026-09-30 review)
- Resolved: the
e2e-prewarmparagraph described--version/--build-dateas existing; it now says they do not exist yet and points at ROCMAI-430. - Resolved: the per-PR row now includes the
[release, nightly]channel legs ofe2e-gpu-strix-ubuntu. - Still stands (non-blocking, reworded): "cannot be silently undone" became "requires the job keys, so moving a lane back into
ci.ymlfails CI". See the first non-blocking item below. - Resolved in spirit: the pasted
effective_serve_enginesource was flagged as a decision; it is still pasted, see the second non-blocking item.
Blocking
- (new) A newly added citation of a tracked source file is a bare backtick, against the AGENTS.md citation rule —
docs/ci-hardware-testing.md:52(`therock.rs`'s index-version parsers are private)
AGENTS.md §5 says to cite a specific repo file "as a link relative to the citing file … not as a bare backtick path", becausedocs-linkscannot see a bare path go stale. The change itself applies the rule toworkflow_contract.rs,serve_cmd.rsandcapability.rs. Thetherock.rsmention is a tracked file and is not in any of the rule's exemptions.apps/rocm/src/therock.rsholds the privateparse_simple_index_versions;tests/e2e-cucumber/tests/e2e/therock_steps.rsis a different file, so the bare name is also ambiguous.
Confidence 85 · mechanical · Fix: link it as[therock.rs](../apps/rocm/src/therock.rs), and nameparse_simple_index_versions.
Non-blocking
- (standing) The guard test is described more strongly than it behaves —
docs/ci-hardware-testing.md:22-26
The text says the test requires the lanes "to appear as job keys … so a PR that moves one of these lanes back intoci.ymlfails CI".self_hosted_workflow_owns_the_gpu_lanes(xtask/src/workflow_contract.rs:755-773) only doessh.contains("e2e-gpu:")on the raw file text. A comment orneeds:mention satisfies it, and a duplicate lane left inci.ymlis not checked. Deleting the job frome2e-selfhosted.ymldoes fail. It also does not covere2e-wsl.
Confidence 70 · mechanical · Fix: say "must contain the job names as text", or leave the claim out. - (standing) The
effective_serve_enginefunction body is copied into the doc —docs/ci-hardware-testing.md:228-238
It matches the source today (tests/e2e-cucumber/src/capability.rs:46-57), but a verbatim copy goes stale silently. The prose already describes the rule and the function is linked.
Confidence 70 · mechanical · Fix: drop the block and keep the prose and the link.
Decisions for the author
- The pending-work note (ROCMAI-430, the pin and the
sdk_versionmatrix) appears twice — tradeoff
It is stated in full in the ladder section and again, shortened, in the pre-warm section. Repetition is a drift risk, but the pre-warm reader benefits from seeing it where the unpinned command lives. Confirm it is deliberate. - The doc records PR numbers and merge state ("merged in PR #415") — tradeoff
Accurate now, and the wording avoids "pending PR" claims, but it will age. Naming the trigger by what the workflow contains would be more durable.
Positive signals
- The claim list was checked against the tree and held up: the nightly schedule, the
release/**push trigger, the(slug, channel)report column identity (crates/e2e-report/src/consolidated.rs:530-541), the Strix/WSL gfx1151 lemonade-only behaviour, and theselect_serve_engineprecedence including the recipe-support gate. - The
select_serve_enginedescription is now accurate and no longer collapses the explicit override and configured default.
Deployment notes
None
What this covered
Every changed line of docs/ci-hardware-testing.md, read against prw-base...a7f1121775aa7f5b5e162b65f66f0f2e3e1e10a3, with each factual claim cross-checked against the workflows, xtask, e2e-report, serve_cmd.rs, therock.rs, capability.rs, host_gpu.rs and AGENTS.md §5. I re-confirmed the blocking item and the first non-blocking item in the checkout before posting. CI on this head is green apart from the self-hosted lanes, which were skipped on this PR (statusCheckRollup).
Did not run: tests and the link check (docs-only); the PR description and ROCMAI-431 were not retrievable, so claims-versus-change was checked against commit messages only. Not reconciled against any discussion here, and I did not read the author's replies when forming the findings.
|
Fixed in 1de88fe: linked the |
1de88fe to
201409b
Compare
tomastola
left a comment
There was a problem hiding this comment.
Re-review at 201409b (last review a7f1121). One docs file. Read in full against the workflows, xtask, e2e-report, serve_cmd.rs, capability.rs and AGENTS.md. Not reconciled against any discussion.
Earlier points
- Resolved: the
e2e-prewarmtense; the per-PR row now mentions the Strix[release, nightly]legs; the baretherock.rscitation is now a link (201409b). - Still open, now blocking: the guard-test claim (inline at :26).
- Still open, non-blocking: the pasted
effective_serve_enginebody (:228-239).
Blocking
- Lane counts will be wrong once this lands —
docs/ci-hardware-testing.md:36-38(inline at :36). Merge-base is 2844d79;origin/mainalready hase2e-gpu-w7900ande2e-gpu-nightly-w7900(#454) and its copy of this doc has W7900 rows. The merge is textually clean, so the ladder would say "4" / "the same 4 platforms" against a Platforms table with more. The workflows on main govern. Fix: rebase, and drop the counts ("the self-hosted jobs in the Platforms table"). Even at this head, "4 jobs in the Platforms table" is off, since the table has 5 rows including theci.ymlmock lane.hardware_testing_docs_cover_all_self_hosted_platformspins that table, so keep it green when rebasing. - The description misstates the change. It says the no-pin fact is "stated once, in this section" and that the pre-warm addendum cross-references rather than re-explains; the doc states the no-pin fact and ROCMAI-430 at
:46-52and again at:365-368. It also says "rebased onto current main" (it is not) and "6 platforms" (7 on main). Either rebase and trim the duplicate, or narrow the description.
Non-blocking
- The artifact-name paragraph (
:199-212) says onlynightly.ymllanes append the channel, and:186-189says two columns per platform only for nightly. Per-PRe2e-gpu-strix-ubuntualso uploadse2e-gpu-strix-ubuntu-${{ matrix.channel }}-report(e2e-selfhosted.yml:410-411,:644), which contradicts the new ladder text at :36. "Silently overwriting" is also loose: before ROCMAI-429 the runs merged into one column (consolidated.rs:531-534). @requires-engineskips only when the scenario is also@requires-gpu(expectation.rs:602);:253-255omits that condition.- The doc presents the harness's engine prediction as the product's rule. The product maps
gfx125*and othergfx90*targets to-dcgpu(vLLM); the harnessnormalize_familydoes not (host_gpu.rs:107-146,capability.rs:75-97). No current lane is affected. :182sayse2e-gpu-rad3/e2e-gpu-mi350p; the job keys aree2e-gpu-nightly-rad3/e2e-gpu-nightly-mi350p.:333-337"every other lane is release-only" is wrong fornightly.yml, which passes--channel ${{ matrix.channel }}on all six lanes.- ROCMAI-431 asks for the
@requires-engine:vllm"silently vanish" trap to be explicit, and for each axis (OS, HW, channel) to be mapped to its stage; both are only implied. - Time-relative prose ("merged in PR #415", "today", "at HEAD"), the 06:00 UTC cron and the pasted snippet will go stale; state the invariant once and link.
Positive signals
- Every added relative link resolves and follows AGENTS.md §5.
- The
select_serve_engineprecedence and the Strix / MI300X conclusions match the code.
CI at this head: everything that ran is green; the self-hosted GPU lanes, Sphinx and skillscope were skipped.
| `self_hosted_workflow_owns_the_gpu_lanes` test requires `e2e-gpu`, | ||
| `e2e-gpu-strix-ubuntu`, and `e2e-gpu-strix-windows` to appear as job keys in | ||
| `e2e-selfhosted.yml`, so a PR that moves one of these lanes back into | ||
| `ci.yml` fails CI rather than merging unnoticed. |
There was a problem hiding this comment.
Blocking: this credits the wrong test. self_hosted_workflow_owns_the_gpu_lanes does sh.contains("e2e-gpu:") and the two sibling strings on the raw text of e2e-selfhosted.yml (workflow_contract.rs:755-774). It never reads ci.yml, so a lane copied into ci.yml with the original left in place still passes, and a comment mentioning the name satisfies it. The code governs. Say what it checks (the three job names appear in e2e-selfhosted.yml), and credit ci_yml_schedules_no_self_hosted_job for keeping self-hosted runs-on labels out of ci.yml.
|
|
||
| | Rung | Workflow | Trigger | Lanes | Blocks a merge/tag? | | ||
| |---|---|---|---|---| | ||
| | Per-PR smoke gate | `e2e-selfhosted.yml` | `push`/`pull_request`/`merge_group` (see "Triggers") | The 4 jobs in the Platforms table below (`e2e-gpu-strix-ubuntu` runs both its `[release, nightly]` channel legs per PR, ROCMAI-125) | No — absent from the required-status-check list | |
There was a problem hiding this comment.
Blocking: "4" will be wrong after rebase. origin/main already has e2e-gpu-w7900 (#454) and its nightly counterpart; this branch's merge-base predates them, so the merge is clean but the counts here and in the next two rows ("the same 4 platforms", "the same 4 per-PR lanes") would miscount. Rebase and drop the numbers. At this head the Platforms table already has 5 rows, so "4 jobs in the Platforms table" is only right for the e2e-selfhosted.yml rows.
Add the missing framing to docs/ci-hardware-testing.md: the three self-hosted GPU gates (per-PR smoke, nightly coverage, release-candidate regression) and how they differ; the validation axes (OS, hardware, channel, SDK-version); why the serve engine isn't an independently selectable matrix axis (quotes effective_serve_engine()); and a short addendum on e2e-prewarm's --version/--build-date pin flags. No lane, artifact, or job-order changes, so every contract-tested table/list/ sentence in the doc is left byte-identical. ROCMAI-431 Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Review found the --version/--build-date paragraph claiming present-tense availability for a flag pair that isn't in this tree yet (ships in #464, stacked on #415, neither merged) — contradicted the ladder section's own "ships in #464" framing two paragraphs up. - e2e-prewarm --version/--build-date: reworded to future tense, matching the ladder section. - Per-PR row: note e2e-gpu-strix-ubuntu already runs both channel legs per PR, not just 4 flat jobs. - "cannot be silently undone": reworded to match what self_hosted_workflow_owns_the_gpu_lanes actually checks (job keys present as text, not a structural pin). Left the two open-decisions-for-the-author items untouched per review (non-defects). Signed-off-by: Juho Vainio <juho.vainio@amd.com>
PR #415's release/** trigger merged 2026-10-01; e2e-selfhosted.yml on main already triggers on push: branches: [main, "release/**"]. Only #464 (the SDK pin) is still pending. Fixes the two passages in the ladder section and the pre-warm section that still said neither PR was merged. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…rose pr-review-watcher flagged 3 blocking issues at f679385: - the engine section said rocm serve picks its engine via the harness's effective_serve_engine(), when capability.rs's own header says that function is the re-implementation and select_serve_engine (main.rs) is the product original. - "the engine is not selectable independent of hardware and OS" overstated it: select_serve_engine checks an explicit --engine and a configured default_engine before falling back to the GPU-family preference, and a scenario can pin either side with @requires-engine:<vllm|lemonade>. - the #464 merge-status prose (two copies) asserted "still pending as of this writing", which goes false the moment #464 merges. Reworded both to state what the tree supports today instead of PR status. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
- rebase onto main (PR #415/#460/#463 already merged there) - cite select_serve_engine's real location (serve_cmd.rs, not main.rs) and turn the three new bare-backtick file citations into relative markdown links per AGENTS.md's citation rule - drop the sentence implying e2e-prewarm already has --version/--build-date flags; state once (in the ladder section) that the pin is pending work (ROCMAI-430) and cross-reference it from the pre-warm addendum instead of repeating it - describe select_serve_engine's full precedence (recipe-support gate, recipe's own preferred engine, platform default), not just the two-step simplification - say 'native Windows' for the lemonade-only engine case; WSL reports os_family linux and stays vLLM-eligible Signed-off-by: Juho Vainio <juho.vainio@amd.com>
A bare backtick path doesn't get checked by docs-links, so it can go stale silently. Link it relative to this file, and name the actual private function instead of just the file. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Rebase picks up #454's W7900 lane (PR #415/#454/#463 all now merged). Drop the hardcoded '4 jobs'/'4 platforms' counts from the validation ladder table so they can't drift again as lanes are added, and credit ci_yml_schedules_no_self_hosted_job alongside self_hosted_workflow_owns_the_gpu_lanes for what each test actually checks. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
201409b to
a167000
Compare
Rebased onto main (picks up #454's W7900 lane). Fixed: lines 22-26 now credit both ci_yml_schedules_no_self_hosted_job (keeps self-hosted runs-on out of ci.yml) and self_hosted_workflow_owns_the_gpu_lanes (checks the three job names appear in e2e-selfhosted.yml). Dropped the hardcoded '4' counts from the ladder table (now 'the self-hosted jobs in the Platforms table below' etc). PR description's nightly-platform count and rebase note updated to match. cargo fmt/clippy/test all clean.
The pre-warm section restated the no-pin/ROCMAI-430 fact that the validation-ladder section already explains, contradicting the PR's own claim that it's stated once. Cross-reference instead of repeating it. Signed-off-by: Juho Vainio <juho.vainio@amd.com>
|
Addressed both blocking findings from the latest review (201409b), across
Also trimmed the duplicated no-pin/ROCMAI-430 explanation (stated once in the ladder section, the pre-warm addendum now just cross-references it instead of re-stating it) and brought the PR description back in sync with the actual rebase state.
|
In plain English
Documentation only, no code changes. Explains the three levels of automated testing we now have (a quick check on every pull request, a deeper check every night, and a full regression check before a release), so it is clear what each one covers.
Summary
Adds the missing framing to
docs/ci-hardware-testing.md:e2e-selfhosted.yml), the nightly coverage gate (nightly.yml, now covering every self-hosted platform plus R9700/rad3 and MI350P, ROCMAI-125/ROCMAI-429), and the release-candidate regression gate (e2e-selfhosted.ymlonrelease/**, ROCMAI-120/EAI-8761). States plainly thatcargo xtask e2e-prewarmhas no--version/--build-datepin today, and that wiring a version pin into the release-branch trigger is pending work (ROCMAI-430) — this is stated once, in this section.workflow_contract.rs'sself_hosted_workflow_owns_the_gpu_lanespinning the three per-PR job names by name, and its siblingci_yml_schedules_no_self_hosted_jobkeeping self-hostedruns-onlabels out ofci.yml(EAI-7548 rationale).(platform_slug, channel)column keying (ROCMAI-429).effective_serve_engine()fromcapability.rs, explaining why every Strix lane is lemonade-only ande2e-gpu(MI300X) is the only per-PR vLLM lane, with an accurate description ofselect_serve_engine's (serve_cmd.rs) real precedence: explicit--engine→ configured default → GPU-family preference (only if the recipe supports that engine) → the recipe's own preferred engine → the platform default. Also corrects "Any Windows host" to "native Windows host", since WSL reportsos_family"linux"and stays vLLM-eligible.No lane, artifact, or job-order changes in this PR, so every contract-tested table/list/sentence in the doc (
hardware_testing_docs_cover_all_self_hosted_platforms) is left byte-identical — verified by running the full xtask suite before and after.Rebased onto current
mainnow that #415, #454 (adding the W7900 lane), and #463 have merged. Every factual claim above was re-verified directly against the current workflow YAML and Rust source after rebasing, including theselect_serve_engineprecedence and thee2e-prewarmCLI surface (xtask/src/main.rs), and all three newly-added source citations now use relative markdown links perAGENTS.md§5.Verification
cargo fmt --all -- --checkclean.cargo test -p xtask --all-targets: 247/0/0/0, includinghardware_testing_docs_cover_all_self_hosted_platforms.grep -rn 'e2e-gpu|strix|mi350p|rad3' docs/ README.mdoutside this file: no hits (no duplicated lane claims elsewhere).No Gherkin scenario: this PR only edits documentation prose, no product or test behavior changes.