Repository navigation
ci(wiki): name each lane's GFX target and package source on the support matrix (ROCMAI-186) - #994
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
jussielo-amd
left a comment
There was a problem hiding this comment.
One blocker, in the new docs paragraph (inline comment): the "reproduce locally" command uses a report anchor with -n, which matches scenario names, so it selects nothing. The rest of the change looks right: the registry field names (index_url, source_layout_generation, wheel_composition.rocm_sdk_target) match runtime_registry.rs, the renderer change is small, and CI on this head is green. This was a static review: nothing was built or run (no Rust toolchain here).
Non-blocking:
docs/ci-hardware-testing.md:208-211: "A row without them has no managed runtime, or its runtime's record predates the fields" is incomplete. A tarball runtime has a record butindex_url: None(therock.rs), and a wheel runtime whose device target is notExactrecords norocm_sdk_target(therock/resolve/wheel.rs:~647), so those rows also lack the fields. Likewise "a lane running on a legacy ... source shows that source" does not hold for a legacy record with noindex_url.tests/e2e-cucumber/src/capability.rs:345: the harness reads onlywheel_composition.rocm_sdk_target. The CLI's ownwheel_composition_device_target(runtime_registry.rs:325) falls back to thedevice-<target>extra inpackage_specs, so a runtime installed with a family label would show no target in a matrix whose title promises each lane's GFX target. Mirror the fallback, or state the limit in the docs.docs/architecture.md:404: theconsolidated/sidecars.rsentry listsPlatformVersionswith itssummary/platform_stackstrings; the newprovenancestring and thegfx_target/sourcefields are not mentioned (AGENTS.md section 5 asks entries to stay true).crates/e2e-report/src/consolidated/sidecars.rs:99-110:provenance()repeats the filter/flatten/join body ofsummary()andplatform_stack(); a small shared join helper would remove it. Alsocapability.rs:296-298: the doc comment's first sentence is spliced into one over-long line; re-wrap.
🤖 by agenthub on AMD AgentHub
| (`rocm install sdk --channel release|nightly --family <family>`) and run the | ||
| scenario with `cargo xtask e2e -- -n "<scenario id>"`; the scenario ids are the | ||
| anchors in the report. |
There was a problem hiding this comment.
Blocking: this command selects nothing. The anchors in the report (##### <id>, markdown.rs:312) are the @id: slugs (for example serve-reuse-exempt-from-no-gpu-refusal), but -n matches the scenario name (serve-28 - Reuse succeeds on a GPU-less host), not the tag, so cargo xtask e2e -- -n "<scenario id>" runs zero scenarios and can look like a clean pass. docs/testing.md (~line 2984) already warns about exactly this. Suggested fix: tell the reader to pass the <key>-<NN> name prefix shown under each anchor (for example -n "serve-28"), or drop the sentence and link docs/testing.md. While here, --family <family> does not take the exact target the cell now shows (gfx1100) and the command omits --version, so it may not install the same runtime.
There was a problem hiding this comment.
Fixed in fb9fe92. The paragraph now tells the reader to install the runtime the Platform cell names (--family/--version to match) and to run the scenario by its name prefix (-n "serve-28"), noting that the report anchors are @id: slugs -n does not match.
Non-blocking items from the review summary, also addressed in that commit:
- the docs now list the cases where a row omits the target or source (tarball runtime, non-exact device target, legacy record with no index URL);
- the harness falls back to the
device-<target>extra inpackage_specs, with a test; provenance()shares ajoin_partshelper with the other two display strings;- the
docs/architecture.mdentry namesprovenance, and thecapability.rsdoc comment is re-wrapped.
jussielo-amd
left a comment
There was a problem hiding this comment.
Thanks, the earlier points are addressed: the reproduce command now selects a scenario by name, the docs list the cases where a row has no target or source, the harness falls back to the device-<target> extra with a test, the architecture entry names provenance, and the join helper is shared. CI on this head is green. One blocker remains, in the same paragraph (inline): the example scenario is a GPU-less one. This was a static review; nothing was built or run (no Rust toolchain here).
Non-blocking:
crates/e2e-report/src/consolidated/html.rs:165still renders the column heading fromversions.summary(), so only the Markdown matrix shows the target and source; the HTML report differs. Either addprovenance()there or say indocs/architecture.mdthat the HTML report omits it.tests/e2e-cucumber/src/capability.rs(the provenance branch, ~366-376): a record withindex_urlbut nosource_layout_generationis not covered by any test, and nothing pins the precedence when bothrocm_sdk_targetand thedevice-extra are present. A url-only record and a both-fields record would pin them.- The PR description is a little stale: it says two new tests and 78/24 passing, the head adds a third (
gfx_target_falls_back_to_the_device_extra_in_package_specs).
🤖 by agenthub on AMD AgentHub
| To reproduce a lane's result locally, install the runtime the Platform cell | ||
| names (`rocm install sdk --channel release|nightly`, adding `--family` or | ||
| `--version` to match it) and run the scenario by its name prefix, for example | ||
| `cargo xtask e2e -- -n "serve-28"`. The anchors in the report are `@id:` slugs, |
There was a problem hiding this comment.
Blocking: the example scenario is the one that cannot reproduce a GPU lane. serve-28 is @requires-no-gpu ("Reuse succeeds on a GPU-less host", model_serving.feature), and -n bypasses the capability gating (docs/testing.md, the serve-28..32 section: run on a host whose GPU state does not match, they "silently report the wrong answer"). This paragraph tells the reader to install the runtime the Platform cell names (a GPU target such as gfx1100) and then run -n "serve-28" on that host, which is exactly the mismatch testing.md warns about. The -n syntax itself is right now; the choice of example is not.
Suggested fix: use a scenario that runs on GPU lanes (for example a real-serve one such as serve-05), and add one clause that -n skips the host capability checks, or link the testing.md section that says so.
There was a problem hiding this comment.
Fixed in e390813.
- The example now uses
serve-05(a@requires-gpuvLLM scenario) instead of the GPU-lessserve-28. - The paragraph says
-nis a regex over the scenario name (not a prefix, not the@id:slug), that it skips the host capability checks, and that the Platform cell's target and source are what to compare against, not what to pass (--familytakes a family label; no flag selects the package source).
Also from the automated review on the same head:
- The harness's
sourcenow has the CLI's two-level fallback towheel_composition.source_layout_generation, with a test that fails without it. docs/architecture.mdnotes the HTML report does not render the target/source line.- The PR body's Testing section now names all three new tests and the current counts.
Left as is: the harness mirroring wheel_composition_device_target by hand. The CLI helper is pub(crate) to apps/rocm, and exposing it would cross the crate-edges boundary for a CI reporting field; the doc comment flags the mirror.
|
|
||
| To reproduce a lane's result locally, install the runtime the Platform cell | ||
| names (`rocm install sdk --channel release|nightly`, adding `--family` or | ||
| `--version` to match it) and run the scenario by its name prefix, for example |
There was a problem hiding this comment.
Non-blocking: two small accuracy points in this sentence. -n is a regex over the scenario name, not a "name prefix" (-n "serve-2" would also match serve-28). And --family takes a family label (gfx110X-all, apps/rocm/src/cli.rs), while the Platform cell now shows an exact target (gfx1100) and an index URL; no flag selects the package source, so "to match it" cannot be fully done. One clause saying the target and source are what to compare against, not what to pass, would be enough.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head fb9fe920
Review — needs work
Full review of the whole change (merge-base 227968dcb30edd574f0405c523536e91f14d4524 against head fb9fe92079c799f3e95bf242428773861310d4d8). Title/body name an internal tracker ID (ROCMAI-186); per policy that ticket was not retrieved and is not discussed here — this change is judged only against what its own PR description states.
Blocking
-
The new "reproduce a lane locally" doc example uses a GPU-less scenario, contradicting the paragraph it's in —
docs/ci-hardware-testing.md:217-222
The paragraph exists to let a reader reproduce a GPU lane's exact target/source (the Platform cell this PR adds). Its example iscargo xtask e2e -- -n "serve-28", butserve-28(tests/e2e-cucumber/features/model_serving.feature:440) isScenario: serve-28 - Reuse succeeds on a GPU-less host, tagged@requires-no-gpu— it is the regression test for the no-GPU reuse exemption and has nothing to do with an installed runtime's GFX target. Following the doc's own instructions with its own example cannot reproduce what the paragraph is about.
Confidence 90 · mechanical · Fix: point at a scenario that actually requires a GPU/managed runtime instead ofserve-28. -
PR description omits one of the three tests the head actually adds — PR body "Testing" section vs. the diff
The description says "two new tests:reports_the_active_runtimes_target_and_source_from_its_record(harness) andmatrix_cell_names_the_exact_target_and_package_source(renderer)" and cites 78/24 passing. The diff at this head also adds a third test,gfx_target_falls_back_to_the_device_extra_in_package_specs(tests/e2e-cucumber/src/capability.rs), which is unmentioned and whose addition should also move the cited pass count. Unmentioned work is unreviewed work.
Confidence 85 · mechanical · Fix: update the Testing section to name all three tests and the current pass counts.
Non-blocking
-
The new GFX-target/package-source feature reaches only the Markdown wiki page, not the HTML consolidated report built in the same CI job —
crates/e2e-report/src/consolidated/html.rs:165,221vs.markdown.rs
html.rsnever calls the newPlatformVersions::provenance(); its column headings still call onlysummary()and its Platform cell shows no version info at all. The samecargo xtask e2e-reportinvocation (xtask/src/e2e_report.rs) writes both outputs in one job ande2e-report-nightlyuploads the HTML as thee2e-consolidated-report-nightlyartifact, so a reader of that artifact sees none of what this PR adds.docs/architecture.md's entries forhtml.rs/sidecars.rsdon't disclose the asymmetry. Reasonable given the PR's stated scope (the wiki page), but worth a line indocs/architecture.mdsaying the HTML renderer omits it, or adding it there.
Confidence 78 · architectural -
The harness's
sourcefield lacks the CLI's own two-level fallback for a legacy manifest, and no test exercises the gap —tests/e2e-cucumber/src/capability.rs(active_runtime_install_root) vs.apps/rocm/src/therock/update.rs:462-468(manifest_source_layout)
Production'smanifest_source_layoutfalls back from the top-levelsource_layout_generationtowheel_composition.source_layout_generationfor "manifests written before the field existed" — a state its own testmanifest_layout_survives_older_manifests(update.rs:786-793) exercises directly. The harness reads only the top-level field, so for that same legacy manifest shape (realindex_url, no top-level generation, generation present on the composition) the matrix would print the bare URL with no(generation)suffix where the CLI's own notion of "the stream this runtime came from" has one.
Confidence 78 · logic · Fix: mirror the same two-level fallback, or note in the doc comment that only the top-level field is read.
Decisions for the author
- Duplicated extraction logic between the harness and the CLI — tradeoff
active_runtime_install_root's newgfx_target/sourcereads re-implementapps/rocm/src/runtime_registry.rs'swheel_composition_device_targetlogic by hand (verified byte-for-byte identical today) rather than calling it, because that function ispub(crate)toapps/rocmand the test crate sits across a layering boundary this repo's own guardrails (cargo xtask check-crate-edges) police. The duplication is consciously flagged in the doc comment ("Mirrors the CLI'swheel_composition_device_target"), but nothing ties the two implementations together going forward — a future change to the production parsing (a new extras separator, a renamed field) would not fail any test in this crate. Confirm this is the intended tradeoff versus exposing a shared helper.
Positive signals
gfx_target/sourceextraction incapability.rsis a verified, field-for-field and byte-for-byte faithful mirror ofwheel_composition_device_targetinruntime_registry.rs, including precedence and extras-parsing.- The
join_partsextraction insidecars.rscleanly removes the triplicated flatten/join bodysummary()/platform_stack()/provenance()previously would have had. - Both new fields are
#[serde(default)]in both crates, so oldplatform.json/report.jsonartifacts missing them deserialize cleanly and render no extra line — exercised by tests on both sides.
Deployment notes
None — this only changes CI reporting/test-harness code; no CLI behavior, schema migration, or user-facing production change is involved.
What this covered
Read completely: crates/e2e-report/src/consolidated/markdown.rs, sidecars.rs, html.rs, mod.rs; tests/e2e-cucumber/src/capability.rs; docs/architecture.md; docs/ci-hardware-testing.md; AGENTS.md (full, 512 lines). Cross-checked against production code: apps/rocm/src/runtime_registry.rs, therock.rs, therock/resolve/wheel.rs, therock/update.rs, runtime_adopt.rs; and tests/e2e-cucumber/features/model_serving.feature for the serve-28 scenario. Diff 227968dcb30e…fb9fe92079c7. Three independent fan-out workers ran the code passes (consolidated-crate renderer/model, harness/production-mirror, and the agent-instruction/history/prior-changes/code-comments passes); findings above were consolidated and re-verified directly against the source by the driver, not taken on a worker's word.
CI: all 39 checks on this head completed, all success or skipped (no failures) — read via the PR's check-run list, not re-derived.
Did not run: no cargo build/cargo test was executed (read-only review; all confidence scores reflect source-tracing, capped below the execution-verified tier). Ticket ROCMAI-186 named but not retrieved, per public-repo policy. Not reconciled against this PR's existing review threads — this review was produced blind, independent of any prior round.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head e390813e
Review — needs work
Full review of the whole change (merge-base 227968dcb30edd574f0405c523536e91f14d4524 against head e390813e45c61da2b24ece0e672f145cbaa9b8ae). Title/body name an internal tracker ID (ROCMAI-186); per policy that ticket was not retrieved and is not discussed here. The description's own account of the change (harness reads gfx_target/source from the registry record into platform.json, the matrix cell prints both, docs/ci-hardware-testing.md explains the cell and a repro command, HTML intentionally doesn't render the new line) matches what the diff does — no mismatch found.
Blocking
- The new registry-record fallback logic has untested branches —
tests/e2e-cucumber/src/capability.rs:350-383(thegfx_target/sourceextraction inactive_runtime_install_root), tests atcapability.rs:~1296-1399
This change adds two fallbacks:gfx_targetpreferswheel_composition.rocm_sdk_targetand falls back to adevice-<target>extra inpackage_specs;source's generation prefers the top-levelsource_layout_generationand falls back towheel_composition.source_layout_generation, otherwise rendering the bare URL with no suffix. None of the three new tests sets both sources of either fallback at once, so no test pins which one wins; and no test gives a record anindex_urlwith no generation at any level, so the bare-URL branch (capability.rs:381,None => url.to_owned()) is never exercised. Traced directly: reordering either fallback's two branches, or deleting theNonearm's bare-URL behavior, would not fail any existing test.
Confidence 82 · logic · Fix: add a test with bothrocm_sdk_targetand a differingdevice-<target>extra present (asserts precedence), and one withindex_urlset and no generation anywhere (asserts the bare-URL output).
Non-blocking
-
The docs' and the new code comment's explanation of why a wheel runtime's target is missing doesn't match what the fallback this PR adds actually covers —
docs/ci-hardware-testing.md(the new "row omits the target" paragraph),tests/e2e-cucumber/src/capability.rs:346-349
The doc attributes a missing target to "a wheel runtime whose device target was not resolved exactly," and the code comment attributes it to "a runtime installed with a family label." Tracing production (apps/rocm/src/therock/resolve/wheel.rs:594-648,apps/rocm/src/therock.rsinstall path):rocm_sdk_targetis set whenever the device target resolvedExact, and an install refuses outright before writing a manifest when it resolvesUndetermined— so no registered runtime, family-labeled or not, can have a realdevice-<target>package spec withoutrocm_sdk_targetalso being set, under current code. The one state that actually produces the gap this PR's fallback handles is a manifest written before therocm_sdk_targetfield existed (a true legacy/schema case), not a family label or an unresolved target. Worth re-wording so the explanation matches the fallback being added.
Confidence 75 · logic -
provenance()'s partial-field case (onlygfx_targetknown, or onlysource) has no dedicated test —crates/e2e-report/src/consolidated/markdown.rs:799-835
The one new renderer test covers both-known and both-absent;summary()/platform_stack()have a similar partial-case test (grid_heading_omits_absent_versions) but it doesn't exerciseprovenance(). Lower risk than the harness gap above since the underlyingjoin_partshelper is already covered at other call sites, but this call site's one-of-two-present case is not.
Confidence 75 · logic
Decisions for the author
- Duplicated extraction logic between the harness and the CLI — tradeoff
active_runtime_install_root's newgfx_target/sourcereads re-implementapps/rocm/src/runtime_registry.rs'swheel_composition_device_targetlogic by hand (verified field-for-field identical) rather than calling it, because that function is private toapps/rocmand the test crate sits across a layering boundary this repo's own guardrails police. This is consciously flagged in the code comment, but nothing ties the two implementations together going forward — a future change to the production parsing would not fail any test in this crate, and now covers two fields instead of one. Confirm this is still the intended tradeoff. - The HTML consolidated report does not render the new GFX-target/package-source line, only the Markdown wiki page does — non-blocking-improvement
docs/architecture.mdwas updated in this PR to disclose the asymmetry explicitly, which resolves what would otherwise be a silent gap. Still worth a decision: a reader of the HTML artifact (uploaded by the same CI job) sees none of what this PR adds. Leaving it Markdown-only matches the PR's stated scope and is a reasonable choice, not a defect.
Positive signals
gfx_targetextraction incapability.rsis a verified, field-for-field faithful mirror ofwheel_composition_device_targetinruntime_registry.rs, including fallback order and extras-parsing syntax.- The shared
join_partshelper removes what would otherwise be a third near-identical filter/flatten/join body acrosssummary(),platform_stack()and the newprovenance(). - Both new fields are
#[serde(default)]in both the harness and the renderer crate, soplatform.json/report.jsonartifacts missing them deserialize cleanly and render no extra line — exercised by tests on both sides. - The repro-command example in
docs/ci-hardware-testing.mdnow names a real GPU/managed-runtime scenario (serve-05, tagged@requires-gpu @requires-engine:vllm, "Given a managed runtime is active"), consistent with the paragraph it sits in.
Deployment notes
None — this only changes CI reporting and test-harness code; no CLI behavior, schema migration, or user-facing production change is involved.
What this covered
Read completely: crates/e2e-report/src/consolidated/markdown.rs, sidecars.rs, html.rs; tests/e2e-cucumber/src/capability.rs; docs/ci-hardware-testing.md; the relevant docs/architecture.md entries; AGENTS.md (full, 512 lines); tests/e2e-cucumber/features/model_serving.feature (the serve-0x scenarios). Cross-checked against production code: apps/rocm/src/runtime_registry.rs, therock.rs, therock/resolve/wheel.rs, therock/update.rs, runtime_adopt.rs. Diff 227968dcb30e…e390813e45c6 (merge-base independently verified against the PR's reported base). Three independent fan-out workers ran the code passes (consolidated-crate renderer/model group; harness/production-mirror group; agent-instruction/history/code-comments pass); findings were consolidated and re-verified directly against the source by the driver, not taken on a worker's word.
CI: all 39 checks on this head completed, all success or skipped (no failures) — read via the PR's live check-run list.
Did not run: no cargo build/cargo test was executed (read-only review; confidence scores reflect source-tracing and direct branch-by-branch reasoning, not execution).
Ticket ROCMAI-186 named but not retrieved, per public-repo policy.
A required pre-check for this automated review confirmed no existing automated review at this exact head commit, which incidentally surfaced the text of two earlier rounds (at older commits on this same PR) to the reviewing context. The Step 4 code-review passes above were run by independent workers with no access to that prior text and no knowledge any earlier round existed; all findings were traced independently against the source before being reported. This report was not reconciled against those earlier rounds' conclusions, and at least one finding here is scored differently than it might have been scored before (a test-coverage gap treated as blocking here under this method's literal standard for untested new behavior).
Not a human review. This is an automated code review; please verify findings independently before acting on them.
jussielo-amd
left a comment
There was a problem hiding this comment.
One blocker, in the same docs paragraph as before (inline comment): the "reproduce locally" recipe does not use the runtime it tells you to install. Your earlier two blockers are resolved: -n now selects by scenario name, and the example is serve-05, a GPU/vLLM scenario. The registry field names and the fallback order in capability.rs match the CLI's own, the renderer change is small, and the architecture entries now name provenance and the HTML omission. This was a static review: nothing was built or run (no Rust toolchain here).
Non-blocking:
- CI: "E2E tests (Strix Halo, Ubuntu, release, current)" is red on this head (the
automations-helper-stops-when-no-watcher-is-enabledscenario found the daemon not running); this PR touches no product code, so it looks like a flake, but it needs a re-run. "E2E tests (W7900, current)" is still queued. docs/ci-hardware-testing.md:~222:serve-05is tagged@requires-engine:vllm, but the Strix Halo, R9700 and W7900 lanes serve on lemonade; naming a lemonade scenario too (for exampleserve-07) and the tags to check would make the example fit every row.docs/architecture.md:~404: thehtml.rsentry says the target and source appear "only on the Markdown wiki page", butconsolidated_summary_markdownalso feeds$GITHUB_STEP_SUMMARY, so the step summary shows them too.crates/e2e-report/src/consolidated/markdown.rs:~63-67anddocs/ci-hardware-testing.md:~435: the cell labeltarget <gfx>collides withtarget=<key>(a runtime key) in the update report;GFX <gfx>would avoid the clash.capability.rs: the harness mirrors the CLI'swheel_composition_device_targetandmanifest_source_layoutby hand with only a comment tying them together; a future change to the record shape would not fail any test here. Fine as a tradeoff, but worth a line in the docs.
🤖 by agenthub on AMD AgentHub
| To reproduce a lane's result locally, install a runtime from the same target | ||
| and source the Platform cell names (`rocm install sdk --channel release|nightly`; | ||
| `--family` takes a family label such as `gfx110X-all`, not the exact target, and | ||
| no flag selects the package source, so treat the cell as what to compare | ||
| against, not what to pass), then run a scenario that exercises it, for example | ||
| `cargo xtask e2e -- -n "serve-05"` (`-n` is a regex over the scenario name, not |
There was a problem hiding this comment.
Blocking: following this recipe does not reproduce the lane the Platform cell names. The runtime you install first is not the one serve-05 uses. Its Given a managed runtime is active step (tests/e2e-cucumber/tests/e2e/runtime_steps.rs, setup_active_runtime) only shares a runtime through use_shared_runtimes(), which is a no-op unless E2E_SHARED_RUNTIMES_DIR is set; otherwise each scenario runs in its own isolated ROCM_CLI_DATA_DIR, sees installed: none, and runs a bare rocm install sdk --yes at the default channel (a multi-GiB pull). So a reader reproducing a nightly row silently gets a different install and may think they reproduced the cell.
Suggested fix: say to pre-warm a runtimes tree for the channel and export E2E_SHARED_RUNTIMES_DIR to it (the shared-runtime section further down this file already describes that), or drop the "install a runtime first" step and say the scenario installs its own at the default channel. One clause on the install size and GPU time would also help.
There was a problem hiding this comment.
Fixed in 505ed78.
- The local-repro paragraph now says to pre-warm a runtimes tree and export
E2E_SHARED_RUNTIMES_DIRto it (linking the shared pre-warmed runtime section), and says why: without it each scenario runs in its own isolated data directory and installs its own runtime at the default channel (multi-GiB, occupies the GPU).
Non-blocking items from the review summary, also taken:
- the example names
serve-07for the lemonade rows next toserve-05, with@requires-engine:vllmas the tag to check; - the Platform cell label is now
GFX <gfx>, so it no longer collides withtarget=<key>in the update report; - the
html.rsentry indocs/architecture.mdnow says the target/source appear in the Markdown output (wiki page and step summary); - the docs note that the harness mirrors the CLI's registry lookup by hand, tied only by a comment.
The red Strix Halo E2E check (automations-helper-stops-when-no-watcher-is-enabled) is untouched by this PR; I have not re-run it.
…rt matrix The nightly wiki matrix showed which ROCm, vLLM and lemonade versions a lane served on and its channel, but not the exact GFX target or where the runtime's packages came from. The e2e harness now reads both from the active runtime's own registry record into platform.json, and the matrix's Platform cell prints them. A record that predates either field shows nothing for it. The page header and docs/ci-hardware-testing.md also say how to read a row and how to reproduce a lane locally. Signed-off-by: juhovainio <juho.vainio@amd.com>
Editing nightly.yml made the PR security scan evaluate the whole file, which has existing findings (workflow-level contents: write, a template expansion in the release build). They are unrelated to the matrix change, so the page-header text goes and the cell content stays. Signed-off-by: juhovainio <juho.vainio@amd.com>
The reproduce-locally paragraph pointed -n at report anchors, which are @id: slugs that -n does not match; name the scenario prefix instead and note --family/--version. Complete the list of rows that omit the target or source. The harness now falls back to the device-<target> extra in package_specs like the CLI does. Share the join helper in sidecars.rs and update the architecture entry. Signed-off-by: juhovainio <juho.vainio@amd.com>
…the legacy source fallback Signed-off-by: juhovainio <juho.vainio@amd.com>
… target distinctly The local-repro recipe now says to export E2E_SHARED_RUNTIMES_DIR (otherwise each scenario installs its own runtime), names a lemonade example alongside the vLLM one, and notes the harness mirrors the CLI's registry lookup by hand. The Platform cell labels the target 'GFX' so it no longer reads like the update report's target=<key>. The architecture entry notes the step summary shares the Markdown renderer. Signed-off-by: juhovainio <juho.vainio@amd.com>
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good. The registry field names and fallbacks mirror the CLI's own, the renderer change is small, and CI on this head is green. This was a static review: nothing was built or run (no Rust toolchain here).
My earlier blockers are resolved: -n selects by scenario name, the example is a GPU scenario (serve-05, serve-07 for lemonade rows), and the paragraph now says to pre-warm a runtimes tree and export E2E_SHARED_RUNTIMES_DIR to it.
Non-blocking:
docs/ci-hardware-testing.md(the "reproduce a lane locally" paragraph, ~217-229): it still doesn't say where the install has to land. A plainrocm install sdkwrites to the default data dir, so an exportedE2E_SHARED_RUNTIMES_DIRpointing elsewhere stays empty and each scenario installs its own runtime. Naming the CI form would close it:cargo xtask e2e-prewarm --channel <release|nightly> --prewarm-dir "$D", thenexport E2E_SHARED_RUNTIMES_DIR="$D/data/runtimes"(whate2e-selfhosted.ymldoes around line 333), orROCM_CLI_DATA_DIR=$D/data rocm install sdk ....- Same paragraph: "It skips the host capability checks" has no clear subject; it means
-n. The new "harness reads the record with its own copy of the CLI's lookup" sentence (~210) is one very long line in a wrapped paragraph. tests/e2e-cucumber/src/capability.rs: no test pins which source wins when bothrocm_sdk_targetand a differingdevice-<target>extra are present, or the bare-URL case (index_urlwith no generation at either level);provenance()with only one of the two fields known is also uncovered in the renderer tests.- The PR description is a little stale: it says the cell prints
target <gfx>(the code printsGFX <gfx>), and "a record that predates either field reports none" overstates it, since the harness also falls back to thedevice-<target>extra. GFX gfx942 · source <url> (multi-arch-v2)leaves "(multi-arch-v2)" unexplained; one line in the docs saying it is the source layout generation would help a reader of the wiki page.
🤖 by agenthub on AMD AgentHub
505ed78 to
f64cd0d
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head f64cd0df075e
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
Implements one acceptance-criteria bullet of ROCMAI-186 ("E2E platform manifests record exact OS, ROCm, engine, GFX target, channel, and source provenance"); does not claim to close it, and its own "Not covered" section correctly scopes out the rest (a declared support claim, unsupported-combination statements).
Blocking
- PR description's rendered-format example is stale — PR description, vs.
crates/e2e-report/src/consolidated/sidecars.rs:104
The description says the Platform cell prints`target <gfx> · source <url> (<generation>)`. The shipped code (and its test,matrix_cell_names_the_exact_target_and_package_sourceinmarkdown.rs) renders"GFX {v}", not"target {v}"— confirmed by the test's own assertion:md.contains("GFX gfx942 · source https://..."). The final commit's own message explains why: "The Platform cell labels the target 'GFX' so it no longer reads like the update report's target=<key>" — the label was changed late, and the top-of-PR description text was not updated to match.
Confidence 100 · mechanical · Fix: update the PR description's example toGFX <gfx> · source <url> (<generation>).
Non-blocking
sourcefallback treats an explicit JSONnulldifferently from an absent key, unlike the function it claims to mirror —tests/e2e-cucumber/src/capability.rs:368-384
active_runtime_install_root'ssourceconstruction doesjson.get("source_layout_generation").or_else(|| json.pointer("/wheel_composition/source_layout_generation")).Value::getreturnsSome(&Value::Null)for a key present with a null value, so.or_elseis not invoked in that case — the wheel_composition fallback is skipped andsourceloses the(generation)suffix even when wheel_composition has one. The realmanifest_source_layout(apps/rocm/src/therock/update.rs:463-470) operates on an already-deserializedOption<String>field, where a JSONnulland an absent key both becomeNone, so its.or_elsealways fires in that situation. The comment directly above this code states it is "the same two-level fallback as the CLI'smanifest_source_layout," which isn't quite true for this one shape. No currently-reachable manifest triggers the divergence (every current write path that setswheel_composition's generation also sets the top-level field, so the top-level value is nevernullwhile the fallback would be needed), andInstalledRuntimeManifest'ssource_layout_generation/index_urlfields lackskip_serializing_if, so a realNonehere is always serialized as explicitnullrather than omitted — worth a second look given how central "absent vs. null" is to this exact fallback's correctness.
Confidence 70 · logic · Fix: flatten the top-level lookup through.and_then(Value::as_str)before the.or_else, so a JSONnullcollapses toNonethe same way the deserializedOption<String>does in the real CLI code.
Decisions for the author
- Hand-mirroring the CLI's registry-parsing logic in a black-box test harness — tradeoff
active_runtime_install_rootnow duplicates two more pieces of CLI logic (wheel_composition_device_target's target fallback,manifest_source_layout's generation fallback) by hand, on top of the pre-existingeffective_serve_engine/probe_amd_gpu_countmirrors the module's own header already flags as a drift risk. Unlike those two, which are pinned by comparison tests, nothing here fails loudly if the real registry record's shape changes — the new docs paragraph itself says this is "tied to it only by a comment." That's a reasonable call for a suite that deliberately cannot importrocm-core, and it's disclosed rather than hidden, but it is worth confirming this is the intended long-term maintenance posture rather than a stepping-stone to exposing the value via arocmcommand.
Positive signals
- The new tests (
reports_the_active_runtimes_target_and_source_from_its_record,gfx_target_falls_back_to_the_device_extra_in_package_specs,source_generation_falls_back_to_the_wheel_composition, and the markdown renderer's with/without case) each target one specific fallback path and mirror the exact manifest shapes those paths exist for, rather than one generic happy-path test. - Replacing the old
(String, Option<String>, PathBuf)return tuple with a namedActiveRuntimestruct (capability.rs) removes a positional-field footgun the two new fields would otherwise have made worse.
Deployment notes
None — this only changes CI test-harness and reporting code (no product/CLI behaviour change, no migration).
What this covered
Every changed file read completely at head f64cd0df075eb3d54cf274bd59a4099df7df9d84 against the PR base: crates/e2e-report/src/consolidated/markdown.rs, crates/e2e-report/src/consolidated/sidecars.rs, docs/architecture.md, docs/ci-hardware-testing.md, tests/e2e-cucumber/src/capability.rs. Cross-checked the harness's hand-mirrored registry logic against apps/rocm/src/runtime_registry.rs and apps/rocm/src/therock/update.rs; traced the full producer→writer→consumer→renderer pipeline (capability.rs → tests/e2e-cucumber/tests/e2e.rs → sidecars.rs → markdown.rs, confirming html.rs indeed never calls .provenance()); confirmed scenarios serve-05/serve-07 exist in tests/e2e-cucumber/features/model_serving.feature and match the docs' vLLM/lemonade claims; confirmed docs/testing.md independently documents the same -n capability-check-bypass behaviour the new paragraph describes. All 39 CI checks on this commit completed, all passing or skipped (none failing).
One background worker independently re-reviewed the Rust code group (capability.rs/sidecars.rs/markdown.rs) blind and surfaced the Non-blocking finding above. A second worker, assigned the docs group, failed twice on a shared concurrency limit (one re-spawn attempted per the method's rule); that pass was instead performed by the driver itself and is not independent — flagged per the method's requirement. History (git blame/git log) and prior-PR-comment passes were not run as separate passes. Ticket ROCMAI-186 was retrieved (title/body/state only, per the method — its comment thread was not read); PR description otherwise reconciles cleanly against the diff. Not reconciled against this PR's own review/comment thread.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Summary
The nightly workflow already publishes a CI-generated
E2E-Support-Matrixwiki page (publish-e2e-wiki). It showed pass/fail per lane and channel, with the ROCm, vLLM and lemonade versions and the OS, but not the exact GFX target or where the runtime's packages came from. This closes that gap rather than adding a second page:collect_versions) now reads the active runtime's exact GFX target (wheel_composition.rocm_sdk_target) and package source (index_urlplussource_layout_generation) from its own registry record intoplatform.json. Nothing comes from a workflow label. A record that predates either field reports none.target <gfx> · source <url> (<generation>)under the versions line, when known.docs/ci-hardware-testing.mddescribes the new cell and how to reproduce a lane locally (rocm install sdk ..., then a GPU scenario such ascargo xtask e2e -- -n "serve-05"). The wiki page header is unchanged: editingnightly.ymlmakes the PR security scan evaluate that whole file, which already has findings of its own (workflow-levelcontents: write, a template expansion in the release build).Not covered
e2e-report-nightlyfails (unchanged, already documented), and the Linux GPU runner pools are not picking up nightly lanes today, so the page will carry few GPU rows until that is fixed.Testing
cargo test -p e2e-report(78 passed) andcargo test -p e2e-cucumber --lib capability(26 passed), including three new tests:reports_the_active_runtimes_target_and_source_from_its_record,gfx_target_falls_back_to_the_device_extra_in_package_specsandsource_generation_falls_back_to_the_wheel_composition(harness), andmatrix_cell_names_the_exact_target_and_package_source(renderer, with and without the fields). The HTML report does not render the new line (noted indocs/architecture.md).rocm_sdk_targetpointer fails the harness test; dropping the provenance line from the Platform cell fails the renderer test.cargo clippy -p e2e-report -p e2e-cucumber --all-targets -- -D warnings: clean.main), so the live wiki output is unseen. The newplatform.jsonfields come from the registry record fields as the CLI writes them today (index_url,source_layout_generation,wheel_composition.rocm_sdk_target); I checked them against the manifest struct, not against a CI-produced file.--no-verifyafter the pre-push hook hit two unrelated ComfyUI tests that pass alone; CI is the authoritative run.