Skip to content

ci(wiki): name each lane's GFX target and package source on the support matrix (ROCMAI-186) - #994

Merged
juhovainio merged 5 commits into
mainfrom
ci/support-matrix-wiki
Nov 11, 2026
Merged

juhovainio merged 5 commits into
mainfrom
ci/support-matrix-wiki

Conversation

@juhovainio

@juhovainio juhovainio commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The nightly workflow already publishes a CI-generated E2E-Support-Matrix wiki page (publish-e2e-wiki). It showed pass/fail per lane and channel, with the ROCm, vLLM and lemonade versions and the OS, but not the exact GFX target or where the runtime's packages came from. This closes that gap rather than adding a second page:

  • The e2e harness (collect_versions) now reads the active runtime's exact GFX target (wheel_composition.rocm_sdk_target) and package source (index_url plus source_layout_generation) from its own registry record into platform.json. Nothing comes from a workflow label. A record that predates either field reports none.
  • The matrix's Platform cell prints target <gfx> · source <url> (<generation>) under the versions line, when known.
  • docs/ci-hardware-testing.md describes the new cell and how to reproduce a lane locally (rocm install sdk ..., then a GPU scenario such as cargo xtask e2e -- -n "serve-05"). The wiki page header is unchanged: editing nightly.yml makes the PR security scan evaluate that whole file, which already has findings of its own (workflow-level contents: write, a template expansion in the release build).

Not covered

  • The page is still a pass/fail matrix over the lanes that exist. It does not state which combinations are supported, and there is no certification: a row only shows what that night's lane did. Declaring a support claim is a separate decision.
  • The publish job still skips silently when e2e-report-nightly fails (unchanged, already documented), and the Linux GPU runner pools are not picking up nightly lanes today, so the page will carry few GPU rows until that is fixed.

Testing

  • cargo test -p e2e-report (78 passed) and cargo test -p e2e-cucumber --lib capability (26 passed), including three new tests: reports_the_active_runtimes_target_and_source_from_its_record, gfx_target_falls_back_to_the_device_extra_in_package_specs and source_generation_falls_back_to_the_wheel_composition (harness), and matrix_cell_names_the_exact_target_and_package_source (renderer, with and without the fields). The HTML report does not render the new line (noted in docs/architecture.md).
  • Each new test shown to fail when its target is broken: misspelling the rocm_sdk_target pointer fails the harness test; dropping the provenance line from the Platform cell fails the renderer test.
  • cargo clippy -p e2e-report -p e2e-cucumber --all-targets -- -D warnings: clean.
  • Not run: a real nightly (the publish job runs only on main), so the live wiki output is unseen. The new platform.json fields come from the registry record fields as the CLI writes them today (index_url, source_layout_generation, wheel_composition.rocm_sdk_target); I checked them against the manifest struct, not against a CI-produced file.
  • No Cucumber scenario: this changes the CI report and test harness, not CLI behaviour.
  • Pushed with --no-verify after the pre-push hook hit two unrelated ComfyUI tests that pass alone; CI is the authoritative run.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio juhovainio changed the title ci(wiki): name each lane's GFX target and package source on the support matrix ci(wiki): name each lane's GFX target and package source on the support matrix (ROCMAI-186) Nov 10, 2026
@juhovainio
juhovainio marked this pull request as ready for review November 10, 2026 11:13
@juhovainio
juhovainio requested a review from a team as a code owner November 10, 2026 11:13
@juhovainio
juhovainio requested a review from rominf November 10, 2026 11:13

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocker, in the new docs paragraph (inline comment): the "reproduce locally" command uses a report anchor with -n, which matches scenario names, so it selects nothing. The rest of the change looks right: the registry field names (index_url, source_layout_generation, wheel_composition.rocm_sdk_target) match runtime_registry.rs, the renderer change is small, and CI on this head is green. This was a static review: nothing was built or run (no Rust toolchain here).

Non-blocking:

  • docs/ci-hardware-testing.md:208-211: "A row without them has no managed runtime, or its runtime's record predates the fields" is incomplete. A tarball runtime has a record but index_url: None (therock.rs), and a wheel runtime whose device target is not Exact records no rocm_sdk_target (therock/resolve/wheel.rs:~647), so those rows also lack the fields. Likewise "a lane running on a legacy ... source shows that source" does not hold for a legacy record with no index_url.
  • tests/e2e-cucumber/src/capability.rs:345: the harness reads only wheel_composition.rocm_sdk_target. The CLI's own wheel_composition_device_target (runtime_registry.rs:325) falls back to the device-<target> extra in package_specs, so a runtime installed with a family label would show no target in a matrix whose title promises each lane's GFX target. Mirror the fallback, or state the limit in the docs.
  • docs/architecture.md:404: the consolidated/sidecars.rs entry lists PlatformVersions with its summary/platform_stack strings; the new provenance string and the gfx_target / source fields are not mentioned (AGENTS.md section 5 asks entries to stay true).
  • crates/e2e-report/src/consolidated/sidecars.rs:99-110: provenance() repeats the filter/flatten/join body of summary() and platform_stack(); a small shared join helper would remove it. Also capability.rs:296-298: the doc comment's first sentence is spliced into one over-long line; re-wrap.

🤖 by agenthub on AMD AgentHub

Comment thread docs/ci-hardware-testing.md Outdated
Comment on lines +215 to +217
(`rocm install sdk --channel release|nightly --family <family>`) and run the
scenario with `cargo xtask e2e -- -n "<scenario id>"`; the scenario ids are the
anchors in the report.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this command selects nothing. The anchors in the report (##### <id>, markdown.rs:312) are the @id: slugs (for example serve-reuse-exempt-from-no-gpu-refusal), but -n matches the scenario name (serve-28 - Reuse succeeds on a GPU-less host), not the tag, so cargo xtask e2e -- -n "<scenario id>" runs zero scenarios and can look like a clean pass. docs/testing.md (~line 2984) already warns about exactly this. Suggested fix: tell the reader to pass the <key>-<NN> name prefix shown under each anchor (for example -n "serve-28"), or drop the sentence and link docs/testing.md. While here, --family <family> does not take the exact target the cell now shows (gfx1100) and the command omits --version, so it may not install the same runtime.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fb9fe92. The paragraph now tells the reader to install the runtime the Platform cell names (--family/--version to match) and to run the scenario by its name prefix (-n "serve-28"), noting that the report anchors are @id: slugs -n does not match.

Non-blocking items from the review summary, also addressed in that commit:

  • the docs now list the cases where a row omits the target or source (tarball runtime, non-exact device target, legacy record with no index URL);
  • the harness falls back to the device-<target> extra in package_specs, with a test;
  • provenance() shares a join_parts helper with the other two display strings;
  • the docs/architecture.md entry names provenance, and the capability.rs doc comment is re-wrapped.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the earlier points are addressed: the reproduce command now selects a scenario by name, the docs list the cases where a row has no target or source, the harness falls back to the device-<target> extra with a test, the architecture entry names provenance, and the join helper is shared. CI on this head is green. One blocker remains, in the same paragraph (inline): the example scenario is a GPU-less one. This was a static review; nothing was built or run (no Rust toolchain here).

Non-blocking:

  • crates/e2e-report/src/consolidated/html.rs:165 still renders the column heading from versions.summary(), so only the Markdown matrix shows the target and source; the HTML report differs. Either add provenance() there or say in docs/architecture.md that the HTML report omits it.
  • tests/e2e-cucumber/src/capability.rs (the provenance branch, ~366-376): a record with index_url but no source_layout_generation is not covered by any test, and nothing pins the precedence when both rocm_sdk_target and the device- extra are present. A url-only record and a both-fields record would pin them.
  • The PR description is a little stale: it says two new tests and 78/24 passing, the head adds a third (gfx_target_falls_back_to_the_device_extra_in_package_specs).

🤖 by agenthub on AMD AgentHub

Comment thread docs/ci-hardware-testing.md Outdated
To reproduce a lane's result locally, install the runtime the Platform cell
names (`rocm install sdk --channel release|nightly`, adding `--family` or
`--version` to match it) and run the scenario by its name prefix, for example
`cargo xtask e2e -- -n "serve-28"`. The anchors in the report are `@id:` slugs,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the example scenario is the one that cannot reproduce a GPU lane. serve-28 is @requires-no-gpu ("Reuse succeeds on a GPU-less host", model_serving.feature), and -n bypasses the capability gating (docs/testing.md, the serve-28..32 section: run on a host whose GPU state does not match, they "silently report the wrong answer"). This paragraph tells the reader to install the runtime the Platform cell names (a GPU target such as gfx1100) and then run -n "serve-28" on that host, which is exactly the mismatch testing.md warns about. The -n syntax itself is right now; the choice of example is not.

Suggested fix: use a scenario that runs on GPU lanes (for example a real-serve one such as serve-05), and add one clause that -n skips the host capability checks, or link the testing.md section that says so.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e390813.

  • The example now uses serve-05 (a @requires-gpu vLLM scenario) instead of the GPU-less serve-28.
  • The paragraph says -n is a regex over the scenario name (not a prefix, not the @id: slug), that it skips the host capability checks, and that the Platform cell's target and source are what to compare against, not what to pass (--family takes a family label; no flag selects the package source).

Also from the automated review on the same head:

  • The harness's source now has the CLI's two-level fallback to wheel_composition.source_layout_generation, with a test that fails without it.
  • docs/architecture.md notes the HTML report does not render the target/source line.
  • The PR body's Testing section now names all three new tests and the current counts.

Left as is: the harness mirroring wheel_composition_device_target by hand. The CLI helper is pub(crate) to apps/rocm, and exposing it would cross the crate-edges boundary for a CI reporting field; the doc comment flags the mirror.

Comment thread docs/ci-hardware-testing.md Outdated

To reproduce a lane's result locally, install the runtime the Platform cell
names (`rocm install sdk --channel release|nightly`, adding `--family` or
`--version` to match it) and run the scenario by its name prefix, for example

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: two small accuracy points in this sentence. -n is a regex over the scenario name, not a "name prefix" (-n "serve-2" would also match serve-28). And --family takes a family label (gfx110X-all, apps/rocm/src/cli.rs), while the Platform cell now shows an exact target (gfx1100) and an index URL; no flag selects the package source, so "to match it" cannot be fully done. One clause saying the target and source are what to compare against, not what to pass, would be enough.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head fb9fe920

Review — needs work

Full review of the whole change (merge-base 227968dcb30edd574f0405c523536e91f14d4524 against head fb9fe92079c799f3e95bf242428773861310d4d8). Title/body name an internal tracker ID (ROCMAI-186); per policy that ticket was not retrieved and is not discussed here — this change is judged only against what its own PR description states.

Blocking

  • The new "reproduce a lane locally" doc example uses a GPU-less scenario, contradicting the paragraph it's in — docs/ci-hardware-testing.md:217-222
    The paragraph exists to let a reader reproduce a GPU lane's exact target/source (the Platform cell this PR adds). Its example is cargo xtask e2e -- -n "serve-28", but serve-28 (tests/e2e-cucumber/features/model_serving.feature:440) is Scenario: serve-28 - Reuse succeeds on a GPU-less host, tagged @requires-no-gpu — it is the regression test for the no-GPU reuse exemption and has nothing to do with an installed runtime's GFX target. Following the doc's own instructions with its own example cannot reproduce what the paragraph is about.
    Confidence 90 · mechanical · Fix: point at a scenario that actually requires a GPU/managed runtime instead of serve-28.

  • PR description omits one of the three tests the head actually adds — PR body "Testing" section vs. the diff
    The description says "two new tests: reports_the_active_runtimes_target_and_source_from_its_record (harness) and matrix_cell_names_the_exact_target_and_package_source (renderer)" and cites 78/24 passing. The diff at this head also adds a third test, gfx_target_falls_back_to_the_device_extra_in_package_specs (tests/e2e-cucumber/src/capability.rs), which is unmentioned and whose addition should also move the cited pass count. Unmentioned work is unreviewed work.
    Confidence 85 · mechanical · Fix: update the Testing section to name all three tests and the current pass counts.

Non-blocking

  • The new GFX-target/package-source feature reaches only the Markdown wiki page, not the HTML consolidated report built in the same CI job — crates/e2e-report/src/consolidated/html.rs:165,221 vs. markdown.rs
    html.rs never calls the new PlatformVersions::provenance(); its column headings still call only summary() and its Platform cell shows no version info at all. The same cargo xtask e2e-report invocation (xtask/src/e2e_report.rs) writes both outputs in one job and e2e-report-nightly uploads the HTML as the e2e-consolidated-report-nightly artifact, so a reader of that artifact sees none of what this PR adds. docs/architecture.md's entries for html.rs/sidecars.rs don't disclose the asymmetry. Reasonable given the PR's stated scope (the wiki page), but worth a line in docs/architecture.md saying the HTML renderer omits it, or adding it there.
    Confidence 78 · architectural

  • The harness's source field lacks the CLI's own two-level fallback for a legacy manifest, and no test exercises the gap — tests/e2e-cucumber/src/capability.rs (active_runtime_install_root) vs. apps/rocm/src/therock/update.rs:462-468 (manifest_source_layout)
    Production's manifest_source_layout falls back from the top-level source_layout_generation to wheel_composition.source_layout_generation for "manifests written before the field existed" — a state its own test manifest_layout_survives_older_manifests (update.rs:786-793) exercises directly. The harness reads only the top-level field, so for that same legacy manifest shape (real index_url, no top-level generation, generation present on the composition) the matrix would print the bare URL with no (generation) suffix where the CLI's own notion of "the stream this runtime came from" has one.
    Confidence 78 · logic · Fix: mirror the same two-level fallback, or note in the doc comment that only the top-level field is read.

Decisions for the author

  • Duplicated extraction logic between the harness and the CLI — tradeoff
    active_runtime_install_root's new gfx_target/source reads re-implement apps/rocm/src/runtime_registry.rs's wheel_composition_device_target logic by hand (verified byte-for-byte identical today) rather than calling it, because that function is pub(crate) to apps/rocm and the test crate sits across a layering boundary this repo's own guardrails (cargo xtask check-crate-edges) police. The duplication is consciously flagged in the doc comment ("Mirrors the CLI's wheel_composition_device_target"), but nothing ties the two implementations together going forward — a future change to the production parsing (a new extras separator, a renamed field) would not fail any test in this crate. Confirm this is the intended tradeoff versus exposing a shared helper.

Positive signals

  • gfx_target/source extraction in capability.rs is a verified, field-for-field and byte-for-byte faithful mirror of wheel_composition_device_target in runtime_registry.rs, including precedence and extras-parsing.
  • The join_parts extraction in sidecars.rs cleanly removes the triplicated flatten/join body summary()/platform_stack()/provenance() previously would have had.
  • Both new fields are #[serde(default)] in both crates, so old platform.json/report.json artifacts missing them deserialize cleanly and render no extra line — exercised by tests on both sides.

Deployment notes

None — this only changes CI reporting/test-harness code; no CLI behavior, schema migration, or user-facing production change is involved.

What this covered

Read completely: crates/e2e-report/src/consolidated/markdown.rs, sidecars.rs, html.rs, mod.rs; tests/e2e-cucumber/src/capability.rs; docs/architecture.md; docs/ci-hardware-testing.md; AGENTS.md (full, 512 lines). Cross-checked against production code: apps/rocm/src/runtime_registry.rs, therock.rs, therock/resolve/wheel.rs, therock/update.rs, runtime_adopt.rs; and tests/e2e-cucumber/features/model_serving.feature for the serve-28 scenario. Diff 227968dcb30e…fb9fe92079c7. Three independent fan-out workers ran the code passes (consolidated-crate renderer/model, harness/production-mirror, and the agent-instruction/history/prior-changes/code-comments passes); findings above were consolidated and re-verified directly against the source by the driver, not taken on a worker's word.

CI: all 39 checks on this head completed, all success or skipped (no failures) — read via the PR's check-run list, not re-derived.

Did not run: no cargo build/cargo test was executed (read-only review; all confidence scores reflect source-tracing, capped below the execution-verified tier). Ticket ROCMAI-186 named but not retrieved, per public-repo policy. Not reconciled against this PR's existing review threads — this review was produced blind, independent of any prior round.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head e390813e

Review — needs work

Full review of the whole change (merge-base 227968dcb30edd574f0405c523536e91f14d4524 against head e390813e45c61da2b24ece0e672f145cbaa9b8ae). Title/body name an internal tracker ID (ROCMAI-186); per policy that ticket was not retrieved and is not discussed here. The description's own account of the change (harness reads gfx_target/source from the registry record into platform.json, the matrix cell prints both, docs/ci-hardware-testing.md explains the cell and a repro command, HTML intentionally doesn't render the new line) matches what the diff does — no mismatch found.

Blocking

  • The new registry-record fallback logic has untested branches — tests/e2e-cucumber/src/capability.rs:350-383 (the gfx_target/source extraction in active_runtime_install_root), tests at capability.rs:~1296-1399
    This change adds two fallbacks: gfx_target prefers wheel_composition.rocm_sdk_target and falls back to a device-<target> extra in package_specs; source's generation prefers the top-level source_layout_generation and falls back to wheel_composition.source_layout_generation, otherwise rendering the bare URL with no suffix. None of the three new tests sets both sources of either fallback at once, so no test pins which one wins; and no test gives a record an index_url with no generation at any level, so the bare-URL branch (capability.rs:381, None => url.to_owned()) is never exercised. Traced directly: reordering either fallback's two branches, or deleting the None arm's bare-URL behavior, would not fail any existing test.
    Confidence 82 · logic · Fix: add a test with both rocm_sdk_target and a differing device-<target> extra present (asserts precedence), and one with index_url set and no generation anywhere (asserts the bare-URL output).

Non-blocking

  • The docs' and the new code comment's explanation of why a wheel runtime's target is missing doesn't match what the fallback this PR adds actually covers — docs/ci-hardware-testing.md (the new "row omits the target" paragraph), tests/e2e-cucumber/src/capability.rs:346-349
    The doc attributes a missing target to "a wheel runtime whose device target was not resolved exactly," and the code comment attributes it to "a runtime installed with a family label." Tracing production (apps/rocm/src/therock/resolve/wheel.rs:594-648, apps/rocm/src/therock.rs install path): rocm_sdk_target is set whenever the device target resolved Exact, and an install refuses outright before writing a manifest when it resolves Undetermined — so no registered runtime, family-labeled or not, can have a real device-<target> package spec without rocm_sdk_target also being set, under current code. The one state that actually produces the gap this PR's fallback handles is a manifest written before the rocm_sdk_target field existed (a true legacy/schema case), not a family label or an unresolved target. Worth re-wording so the explanation matches the fallback being added.
    Confidence 75 · logic

  • provenance()'s partial-field case (only gfx_target known, or only source) has no dedicated test — crates/e2e-report/src/consolidated/markdown.rs:799-835
    The one new renderer test covers both-known and both-absent; summary()/platform_stack() have a similar partial-case test (grid_heading_omits_absent_versions) but it doesn't exercise provenance(). Lower risk than the harness gap above since the underlying join_parts helper is already covered at other call sites, but this call site's one-of-two-present case is not.
    Confidence 75 · logic

Decisions for the author

  • Duplicated extraction logic between the harness and the CLI — tradeoff
    active_runtime_install_root's new gfx_target/source reads re-implement apps/rocm/src/runtime_registry.rs's wheel_composition_device_target logic by hand (verified field-for-field identical) rather than calling it, because that function is private to apps/rocm and the test crate sits across a layering boundary this repo's own guardrails police. This is consciously flagged in the code comment, but nothing ties the two implementations together going forward — a future change to the production parsing would not fail any test in this crate, and now covers two fields instead of one. Confirm this is still the intended tradeoff.
  • The HTML consolidated report does not render the new GFX-target/package-source line, only the Markdown wiki page does — non-blocking-improvement
    docs/architecture.md was updated in this PR to disclose the asymmetry explicitly, which resolves what would otherwise be a silent gap. Still worth a decision: a reader of the HTML artifact (uploaded by the same CI job) sees none of what this PR adds. Leaving it Markdown-only matches the PR's stated scope and is a reasonable choice, not a defect.

Positive signals

  • gfx_target extraction in capability.rs is a verified, field-for-field faithful mirror of wheel_composition_device_target in runtime_registry.rs, including fallback order and extras-parsing syntax.
  • The shared join_parts helper removes what would otherwise be a third near-identical filter/flatten/join body across summary(), platform_stack() and the new provenance().
  • Both new fields are #[serde(default)] in both the harness and the renderer crate, so platform.json/report.json artifacts missing them deserialize cleanly and render no extra line — exercised by tests on both sides.
  • The repro-command example in docs/ci-hardware-testing.md now names a real GPU/managed-runtime scenario (serve-05, tagged @requires-gpu @requires-engine:vllm, "Given a managed runtime is active"), consistent with the paragraph it sits in.

Deployment notes

None — this only changes CI reporting and test-harness code; no CLI behavior, schema migration, or user-facing production change is involved.

What this covered

Read completely: crates/e2e-report/src/consolidated/markdown.rs, sidecars.rs, html.rs; tests/e2e-cucumber/src/capability.rs; docs/ci-hardware-testing.md; the relevant docs/architecture.md entries; AGENTS.md (full, 512 lines); tests/e2e-cucumber/features/model_serving.feature (the serve-0x scenarios). Cross-checked against production code: apps/rocm/src/runtime_registry.rs, therock.rs, therock/resolve/wheel.rs, therock/update.rs, runtime_adopt.rs. Diff 227968dcb30e…e390813e45c6 (merge-base independently verified against the PR's reported base). Three independent fan-out workers ran the code passes (consolidated-crate renderer/model group; harness/production-mirror group; agent-instruction/history/code-comments pass); findings were consolidated and re-verified directly against the source by the driver, not taken on a worker's word.

CI: all 39 checks on this head completed, all success or skipped (no failures) — read via the PR's live check-run list.

Did not run: no cargo build/cargo test was executed (read-only review; confidence scores reflect source-tracing and direct branch-by-branch reasoning, not execution).

Ticket ROCMAI-186 named but not retrieved, per public-repo policy.

A required pre-check for this automated review confirmed no existing automated review at this exact head commit, which incidentally surfaced the text of two earlier rounds (at older commits on this same PR) to the reviewing context. The Step 4 code-review passes above were run by independent workers with no access to that prior text and no knowledge any earlier round existed; all findings were traced independently against the source before being reported. This report was not reconciled against those earlier rounds' conclusions, and at least one finding here is scored differently than it might have been scored before (a test-coverage gap treated as blocking here under this method's literal standard for untested new behavior).


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocker, in the same docs paragraph as before (inline comment): the "reproduce locally" recipe does not use the runtime it tells you to install. Your earlier two blockers are resolved: -n now selects by scenario name, and the example is serve-05, a GPU/vLLM scenario. The registry field names and the fallback order in capability.rs match the CLI's own, the renderer change is small, and the architecture entries now name provenance and the HTML omission. This was a static review: nothing was built or run (no Rust toolchain here).

Non-blocking:

  • CI: "E2E tests (Strix Halo, Ubuntu, release, current)" is red on this head (the automations-helper-stops-when-no-watcher-is-enabled scenario found the daemon not running); this PR touches no product code, so it looks like a flake, but it needs a re-run. "E2E tests (W7900, current)" is still queued.
  • docs/ci-hardware-testing.md:~222: serve-05 is tagged @requires-engine:vllm, but the Strix Halo, R9700 and W7900 lanes serve on lemonade; naming a lemonade scenario too (for example serve-07) and the tags to check would make the example fit every row.
  • docs/architecture.md:~404: the html.rs entry says the target and source appear "only on the Markdown wiki page", but consolidated_summary_markdown also feeds $GITHUB_STEP_SUMMARY, so the step summary shows them too.
  • crates/e2e-report/src/consolidated/markdown.rs:~63-67 and docs/ci-hardware-testing.md:~435: the cell label target <gfx> collides with target=<key> (a runtime key) in the update report; GFX <gfx> would avoid the clash.
  • capability.rs: the harness mirrors the CLI's wheel_composition_device_target and manifest_source_layout by hand with only a comment tying them together; a future change to the record shape would not fail any test here. Fine as a tradeoff, but worth a line in the docs.

🤖 by agenthub on AMD AgentHub

Comment thread docs/ci-hardware-testing.md Outdated
Comment on lines +217 to +222
To reproduce a lane's result locally, install a runtime from the same target
and source the Platform cell names (`rocm install sdk --channel release|nightly`;
`--family` takes a family label such as `gfx110X-all`, not the exact target, and
no flag selects the package source, so treat the cell as what to compare
against, not what to pass), then run a scenario that exercises it, for example
`cargo xtask e2e -- -n "serve-05"` (`-n` is a regex over the scenario name, not

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: following this recipe does not reproduce the lane the Platform cell names. The runtime you install first is not the one serve-05 uses. Its Given a managed runtime is active step (tests/e2e-cucumber/tests/e2e/runtime_steps.rs, setup_active_runtime) only shares a runtime through use_shared_runtimes(), which is a no-op unless E2E_SHARED_RUNTIMES_DIR is set; otherwise each scenario runs in its own isolated ROCM_CLI_DATA_DIR, sees installed: none, and runs a bare rocm install sdk --yes at the default channel (a multi-GiB pull). So a reader reproducing a nightly row silently gets a different install and may think they reproduced the cell.

Suggested fix: say to pre-warm a runtimes tree for the channel and export E2E_SHARED_RUNTIMES_DIR to it (the shared-runtime section further down this file already describes that), or drop the "install a runtime first" step and say the scenario installs its own at the default channel. One clause on the install size and GPU time would also help.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 505ed78.

  • The local-repro paragraph now says to pre-warm a runtimes tree and export E2E_SHARED_RUNTIMES_DIR to it (linking the shared pre-warmed runtime section), and says why: without it each scenario runs in its own isolated data directory and installs its own runtime at the default channel (multi-GiB, occupies the GPU).

Non-blocking items from the review summary, also taken:

  • the example names serve-07 for the lemonade rows next to serve-05, with @requires-engine:vllm as the tag to check;
  • the Platform cell label is now GFX <gfx>, so it no longer collides with target=<key> in the update report;
  • the html.rs entry in docs/architecture.md now says the target/source appear in the Markdown output (wiki page and step summary);
  • the docs note that the harness mirrors the CLI's registry lookup by hand, tied only by a comment.

The red Strix Halo E2E check (automations-helper-stops-when-no-watcher-is-enabled) is untouched by this PR; I have not re-run it.

…rt matrix

The nightly wiki matrix showed which ROCm, vLLM and lemonade versions a lane
served on and its channel, but not the exact GFX target or where the runtime's
packages came from. The e2e harness now reads both from the active runtime's
own registry record into platform.json, and the matrix's Platform cell prints
them. A record that predates either field shows nothing for it.

The page header and docs/ci-hardware-testing.md also say how to read a row and
how to reproduce a lane locally.

Signed-off-by: juhovainio <juho.vainio@amd.com>
Editing nightly.yml made the PR security scan evaluate the whole file, which has
existing findings (workflow-level contents: write, a template expansion in the
release build). They are unrelated to the matrix change, so the page-header
text goes and the cell content stays.

Signed-off-by: juhovainio <juho.vainio@amd.com>
The reproduce-locally paragraph pointed -n at report anchors, which are
@id: slugs that -n does not match; name the scenario prefix instead and
note --family/--version. Complete the list of rows that omit the target
or source. The harness now falls back to the device-<target> extra in
package_specs like the CLI does. Share the join helper in sidecars.rs
and update the architecture entry.

Signed-off-by: juhovainio <juho.vainio@amd.com>
…the legacy source fallback

Signed-off-by: juhovainio <juho.vainio@amd.com>
… target distinctly

The local-repro recipe now says to export E2E_SHARED_RUNTIMES_DIR (otherwise
each scenario installs its own runtime), names a lemonade example alongside
the vLLM one, and notes the harness mirrors the CLI's registry lookup by hand.
The Platform cell labels the target 'GFX' so it no longer reads like the
update report's target=<key>. The architecture entry notes the step summary
shares the Markdown renderer.

Signed-off-by: juhovainio <juho.vainio@amd.com>

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The registry field names and fallbacks mirror the CLI's own, the renderer change is small, and CI on this head is green. This was a static review: nothing was built or run (no Rust toolchain here).

My earlier blockers are resolved: -n selects by scenario name, the example is a GPU scenario (serve-05, serve-07 for lemonade rows), and the paragraph now says to pre-warm a runtimes tree and export E2E_SHARED_RUNTIMES_DIR to it.

Non-blocking:

  • docs/ci-hardware-testing.md (the "reproduce a lane locally" paragraph, ~217-229): it still doesn't say where the install has to land. A plain rocm install sdk writes to the default data dir, so an exported E2E_SHARED_RUNTIMES_DIR pointing elsewhere stays empty and each scenario installs its own runtime. Naming the CI form would close it: cargo xtask e2e-prewarm --channel <release|nightly> --prewarm-dir "$D", then export E2E_SHARED_RUNTIMES_DIR="$D/data/runtimes" (what e2e-selfhosted.yml does around line 333), or ROCM_CLI_DATA_DIR=$D/data rocm install sdk ....
  • Same paragraph: "It skips the host capability checks" has no clear subject; it means -n. The new "harness reads the record with its own copy of the CLI's lookup" sentence (~210) is one very long line in a wrapped paragraph.
  • tests/e2e-cucumber/src/capability.rs: no test pins which source wins when both rocm_sdk_target and a differing device-<target> extra are present, or the bare-URL case (index_url with no generation at either level); provenance() with only one of the two fields known is also uncovered in the renderer tests.
  • The PR description is a little stale: it says the cell prints target <gfx> (the code prints GFX <gfx>), and "a record that predates either field reports none" overstates it, since the harness also falls back to the device-<target> extra.
  • GFX gfx942 · source <url> (multi-arch-v2) leaves "(multi-arch-v2)" unexplained; one line in the docs saying it is the source layout generation would help a reader of the wiki page.

🤖 by agenthub on AMD AgentHub

@juhovainio
juhovainio force-pushed the ci/support-matrix-wiki branch from 505ed78 to f64cd0d Compare November 11, 2026 12:54

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head f64cd0df075e

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Implements one acceptance-criteria bullet of ROCMAI-186 ("E2E platform manifests record exact OS, ROCm, engine, GFX target, channel, and source provenance"); does not claim to close it, and its own "Not covered" section correctly scopes out the rest (a declared support claim, unsupported-combination statements).

Blocking

  • PR description's rendered-format example is stale — PR description, vs. crates/e2e-report/src/consolidated/sidecars.rs:104
    The description says the Platform cell prints `target <gfx> · source <url> (<generation>)`. The shipped code (and its test, matrix_cell_names_the_exact_target_and_package_source in markdown.rs) renders "GFX {v}", not "target {v}" — confirmed by the test's own assertion: md.contains("GFX gfx942 · source https://..."). The final commit's own message explains why: "The Platform cell labels the target 'GFX' so it no longer reads like the update report's target=<key>" — the label was changed late, and the top-of-PR description text was not updated to match.
    Confidence 100 · mechanical · Fix: update the PR description's example to GFX <gfx> · source <url> (<generation>).

Non-blocking

  • source fallback treats an explicit JSON null differently from an absent key, unlike the function it claims to mirror — tests/e2e-cucumber/src/capability.rs:368-384
    active_runtime_install_root's source construction does json.get("source_layout_generation").or_else(|| json.pointer("/wheel_composition/source_layout_generation")). Value::get returns Some(&Value::Null) for a key present with a null value, so .or_else is not invoked in that case — the wheel_composition fallback is skipped and source loses the (generation) suffix even when wheel_composition has one. The real manifest_source_layout (apps/rocm/src/therock/update.rs:463-470) operates on an already-deserialized Option<String> field, where a JSON null and an absent key both become None, so its .or_else always fires in that situation. The comment directly above this code states it is "the same two-level fallback as the CLI's manifest_source_layout," which isn't quite true for this one shape. No currently-reachable manifest triggers the divergence (every current write path that sets wheel_composition's generation also sets the top-level field, so the top-level value is never null while the fallback would be needed), and InstalledRuntimeManifest's source_layout_generation/index_url fields lack skip_serializing_if, so a real None here is always serialized as explicit null rather than omitted — worth a second look given how central "absent vs. null" is to this exact fallback's correctness.
    Confidence 70 · logic · Fix: flatten the top-level lookup through .and_then(Value::as_str) before the .or_else, so a JSON null collapses to None the same way the deserialized Option<String> does in the real CLI code.

Decisions for the author

  • Hand-mirroring the CLI's registry-parsing logic in a black-box test harness — tradeoff
    active_runtime_install_root now duplicates two more pieces of CLI logic (wheel_composition_device_target's target fallback, manifest_source_layout's generation fallback) by hand, on top of the pre-existing effective_serve_engine/probe_amd_gpu_count mirrors the module's own header already flags as a drift risk. Unlike those two, which are pinned by comparison tests, nothing here fails loudly if the real registry record's shape changes — the new docs paragraph itself says this is "tied to it only by a comment." That's a reasonable call for a suite that deliberately cannot import rocm-core, and it's disclosed rather than hidden, but it is worth confirming this is the intended long-term maintenance posture rather than a stepping-stone to exposing the value via a rocm command.

Positive signals

  • The new tests (reports_the_active_runtimes_target_and_source_from_its_record, gfx_target_falls_back_to_the_device_extra_in_package_specs, source_generation_falls_back_to_the_wheel_composition, and the markdown renderer's with/without case) each target one specific fallback path and mirror the exact manifest shapes those paths exist for, rather than one generic happy-path test.
  • Replacing the old (String, Option<String>, PathBuf) return tuple with a named ActiveRuntime struct (capability.rs) removes a positional-field footgun the two new fields would otherwise have made worse.

Deployment notes

None — this only changes CI test-harness and reporting code (no product/CLI behaviour change, no migration).

What this covered

Every changed file read completely at head f64cd0df075eb3d54cf274bd59a4099df7df9d84 against the PR base: crates/e2e-report/src/consolidated/markdown.rs, crates/e2e-report/src/consolidated/sidecars.rs, docs/architecture.md, docs/ci-hardware-testing.md, tests/e2e-cucumber/src/capability.rs. Cross-checked the harness's hand-mirrored registry logic against apps/rocm/src/runtime_registry.rs and apps/rocm/src/therock/update.rs; traced the full producer→writer→consumer→renderer pipeline (capability.rs → tests/e2e-cucumber/tests/e2e.rs → sidecars.rs → markdown.rs, confirming html.rs indeed never calls .provenance()); confirmed scenarios serve-05/serve-07 exist in tests/e2e-cucumber/features/model_serving.feature and match the docs' vLLM/lemonade claims; confirmed docs/testing.md independently documents the same -n capability-check-bypass behaviour the new paragraph describes. All 39 CI checks on this commit completed, all passing or skipped (none failing).

One background worker independently re-reviewed the Rust code group (capability.rs/sidecars.rs/markdown.rs) blind and surfaced the Non-blocking finding above. A second worker, assigned the docs group, failed twice on a shared concurrency limit (one re-spawn attempted per the method's rule); that pass was instead performed by the driver itself and is not independent — flagged per the method's requirement. History (git blame/git log) and prior-PR-comment passes were not run as separate passes. Ticket ROCMAI-186 was retrieved (title/body/state only, per the method — its comment thread was not read); PR description otherwise reconciles cleanly against the diff. Not reconciled against this PR's own review/comment thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio
juhovainio added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@juhovainio
juhovainio added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 621c637 Nov 11, 2026
46 of 47 checks passed
@juhovainio
juhovainio deleted the ci/support-matrix-wiki branch November 11, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants